An almost identical version of this article was published on Perl.com in August 2005. http://perl.com/pub/a/2005/08/25/tools.html

Perl Needs Better Tools

Copyright © 2005 by Eigenstate Consulting, LLC. All Rights Reserved.
by Matisse Enzer

If you are afraid that Perl may be in danger of becoming irrelevant for medium-to-large projects then read on.

Perl is in danger of becoming a fading language - new programmers are learning Java and Python in college, and companies like Google hardly use Perl at all.

The Scary Part

I have discussed the future of Perl with managers from companies that currently use it and find that they are worried about the future of Perl. One company I spoke with here in San Francisco is rewriting their core application in Java, another company is worried they will not be able to find new Perl programmers down the road, another uses Perl for major projects but suffers from the difficulty in refactoring their extensive code base.

There are many reasons why companies are worried about the future of Perl, and in this article I am offering a part of a solution; better tools for Perl can be a major part of keeping Perl relevant and effective as the primary language for medium and large projects.

When measuring the effectiveness of a development environment (people + language + tools + processes, etc.) a key measure is how expensive and painful it is to make changes to existing code. Once a project or system has grown to thousands of lines of code in dozens (or hundreds) of modules the cost of making changes can escalate to the point where the team is afraid to make any significant change. Excellent tools are one of the ways to avoid this unhappy situation, or at least reduce its impact (other factors are excellent processes and of course excellent people.)

A Partial Answer: 21st Century Integrated Development Environments for Perl

I propose that more, high-quality development tools will help keep Perl relevant and alive in medium and large project environments.

In this article I will focus mainly on what are commonly called IDEs - Integrated Development Environments - primarily IDE's with a graphical interface.

An IDE is an integrated set of tools for programming combining a source code editor with a variety of other tools into a single package. Common features of modern IDE's include refactoring support, version control, real-time syntax checking, auto-completion of code while typing, etc. I'll have more to say about these features later in this article.

I want to make it clear right at the outset that a team of highly skilled Perl programmers, using only tools that have been around for years (e.g. emacs, vi, cvs, make, etc.) can and do build large projects that are very sophisticated and successful. I am not worried about those programmers. I am worried about the larger population of programmers with 1-5 years of experience, and those who have not yet begun to program - the next generation of Perl programmers.

Great tools will not make a bad programmer into a good programmer, but they will certainly make a good programmer better.

The tools for Perl are years behind what is available for other languages, particularly Java.

One powerful example is the lack of graphical IDE's for Perl with excellent support for refactoring - there are several IDE's for Java that have extensive refactoring support and only one for Perl that supports only single refactoring action (The EPIC plug in for Eclipse.)

For an example of how good IDEs have inspired at least one Perl developer see Adam Kennedy's perl.com article on his new PPI module and Scott Sotka's Devel::Refactor module (used in EPIC)

I acknowledge that a graphical IDE is not the be-all of good tools, and just as some writers reject word processors in favor of typewriters or hand-written manuscripts, some programmers reject graphical IDE's and would refuse a job that required them to use one.

Not everyone has (nor should have) the same tool set, and there are things a pencil can do that vi and emacs will never do. That said, IDE's are in fact widely used in businesses doing larger projects, and for many programmers and teams they provide major increases in productivity.

Another important point to make up front here is that while this article discusses over a dozen specific tools or features it is having all the tools in a single package that will produce the by far the biggest value. An IDE that provides all of these features in a single package that can be easily installed, easily extended, and easily maintained across an entire development team has far more value than simply the list of the parts.

There is a big win when the features provided by an IDE immediately upon installation include all or almost all of the tools and features discussed later in this article, and where the features "know" about each other. For example, if you enter the name of a non-existent subroutine and the real-time syntax checker catches this that is good. It is much better if the code-assist feature then pops up a context menu offering: 1) to create a stub for the subroutine, 2) to correct the name to an existing similar subroutine name or method name from another class that is available to the current file. (That example by the way, is standard behavior for some Java IDE's)

What would a 21st-century Perl Tool set Contain?

What Perl needs is a few great IDE's - not just one, but more than one so that people have a diverse set to choose from. And not just good IDE's, Perl deserves and needs a few great IDE's, that lead the pack and set the standard for IDE's in other languages.

I am well aware that the dynamic nature of Perl makes it harder to have a program that can read and understand a Perl program, especially a large and complex one, but:

  1. The difficulty in comprehending a Perl program makes the value of such a tool all the greater, and
  2. I have faith that the Perl community can overcome some of the built-in challenges of Perl, indeed, it is among the greatest strengths of Perl that Perl users are able to adapt the language to their needs.

A great Perl IDE will contain at least the following, plus other features I haven't thought of (and of course there must be many of those!)

Most of the screen shot examples in this article use the EPIC Perl IDE, since at present it has the largest numbers of the features on my list (although it certainly doesn't have all of them.)

Syntax-coloring text editor


Most of you have probably seen this - it is available under vim, emacs, BBEdit, TextPad, just about every decent text editor will colorize source code so that keywords, operators, variables, etc.  each have their own color, making it easier to spot syntax errors like forgetting to close a quote pair.

Real-time Syntax-checking - catch and display syntax errors as you type


Here's an example:

real-time syntax check example

The IDE is showing that line 4 has an error because of the missing ) and line 5 has an error because $naame hasn't been declared (and use strict is in effect.)

A key point here is that the IDE shows these errors right away, before you save and compile the code. (In this example, the EPIC IDE lets you specify how often to run the syntax check from 0.01 to 10.00 seconds of idle time, or only on-demand.)

As nice as this is, even better would be if the IDE also offered ways to fix the problem, for example, offering to change $naame to $name. Here's an example of an IDE that does exactly that, unfortunately it does it for Java, not Perl:

Example of syntax help rom the IDE

It would be great if there were several Perl IDE's that offered this kind of help.

Version control integration - checkout and compare code using CVS, subversion, etc.


Version control software is used by all non-insane large projects these days. The most common version control software systems are probably CVS, Perforce, Subversion, and Visual SourceSafe.

Here's an example of comparing the local version of a file to an older version from the CVS repository:
[PRODUCTION NOTE: Maybe put a thumbnail in the article, linked to the full-size image. --ME]
Example of comparing local file to older version in CVS

CVS integration is available in many modern code editors including emacs, vim, and BBEdit, as well as graphical IDE's such as Eclipse and Komodo Pro. Subversion integration is available as a plug-in for Eclipse; Komodo Pro supports Perforce and Subversion.

Code-assist editor (e.g. get a menu of methods when you type in an object reference.)


Let's say you have just typed in an object reference and want to call a method on the object, but you are not sure what the method name is. Wouldn't it be nice if the editor popped up a menu listing all the methods available for that object? It might look something like this:

Example of automatic code completion

In this example the IDE is able to figure out which class the object $q is an instance of, and goes and figures out the names of the available methods. If you type a p then the list shows only the method names beginning with p, and if you type pa, then only the param() and parse_params() methods would be shown.

Excellent refactoring support


The easier it is to do refactoring the more often it will be done. Here is a list of a few of the most common refactoring operations - your personal list will probably be a little different. All of these are things you can do "manually", the idea is to make them into one or two-click operations so that you will do them much more often. (For a extensive list of refactoring operations, see Martin Fowler's Alphabetical List of Refactorings.)

  1. Extract Subroutine/Method
  2. Rename Subroutine/Method
  3. Rename Variable
  4. Change Subroutine/Method signature (add/remove parameters)
  5. Move Subroutine/Method to different file/class
  6. Change package name
  1. Extract Subroutine/Method

    The IDE should create a new subroutine using the selected code, and replace the selected code with a call to the new subroutine, with the proper parameters. Here's an example of using the Extract Subroutine refactoring from Eclipse/EPIC (which uses the Devel::Refactor module):

    First you select a chunk of code to turn into a new subroutine, and then select "Extract Subroutine" from a context menu. You then get the a dialog box asking for the name of the new subroutine:

    Code before Extract Subroutine refactoring

    The IDE replaces the selected code with a call to the new subroutine, making reasonable guesses about the parameters and return values. You may need to manually clean up the result.

    Code after Extract Subroutine refactoring.

    Here is the new subroutine created by the IDE - in this case it needs to changes, but sometimes you will need to manually adjust the parameters and/or return value(s).

    The new subroutine created by Extract Subroutine

  2. Ideally you should be prompted to replace similar chunks of code with calls to the new subroutine.

  3. Rename Subroutine/Method

    The IDE should find all the calls to the subroutine throughout your project and offer to change them for you. You should be able to see a preview of all the places a change could occur and accept or reject each one on a case-by-case basis. The action should be un-doable.

  4. Rename Variable

    Like Rename Subroutine this feature should find all occurrences throughout the project and offer to make the changes for you.

  5. Change Subroutine/Method signature (add/remove parameters)

    The IDE should be able to make reasonable guesses about whether or not each subroutine or method call is supplying the proper parameters. Partly this is to enable the real-time syntax checking mentioned above,  and partly this is to enable you to select a subroutine declaration and tell the IDE you want to refactor it by adding or removing a parameter. The IDE should then prompt you for the change(s) you want to make and do its best to find all the existing calls to the subroutine and offer to correct the subroutine calls to supply the new parameters. Obviously this is an especially tricky thing to do in Perl where subroutine parameters are fished out of @_ so the IDE would have to look carefully at how shift, @_ and $_[] are treated inside the subroutine code in order to have a reasonable guess about the parameters the subroutine is expecting. In many common cases though, a Perl IDE  could make a reasonable guess about the parameters, such as in the following two examples, so that if you added or removed one, it could immediately prompt you about making corrections throughout the project:

    sub doSomething {
        my $gender = shift;
        my $age = shift;
        # Not too terribly hard to guess that $gender and $age are params
    }

    sub anotherThing {
        my ($speed,$direction) = @_;
        # No magic needed to guess $speed and $direction are params.
    }

  6. Move Subroutine/Method to different file/class

    This refactoring operation should give you a list or dialog box to choose the destination file in your project. The IDE should allow you to preview all the changes that it would make to accomplish the move, which will include updating call to the subroutine/method to use the proper class. At a minimum the IDE should show you or list all the calls to the subroutine so you can make the appropriate changes yourself. Ideally the IDE should make a guess about possible destinations, for example if $self is a parameter to the method being moved then the IDE might try assuming the method is an object (instance) method and initially only list destination classes that inherit from the source class, or which the source class inherits from. 

  7. Change package name

    As with Rename Subroutine and Rename Variable, when changing a package name the IDE should offer to update all existing references throughout your project.


Tree view and navigation of source files and resources

Another useful feature of good IDE's is being able to view all of the code for a project, or multiple projects, in a tree-format, where you can "fold" and "unfold" the contents of folders. All the modern graphical IDE's support this, even with multiple projects in different languages.

Being able to view your project in this manner gives you both a high-level overview, and the ability to drill down into specific files, and to mix levels of detail by having some folders show their contents and some not.

For example, here is a partial screen shot from ActiveState's Komodo IDE:

Tree view of code in Komodo

Support for creating and running unit tests

Anyone who has installed Perl modules from CPAN has seen Unit Tests - these are the various, often copious tests that run when you execute the make test part of the installation process. The vast majority of CPAN modules include a suite of tests, often using the Test::Harness and/or Test::More modules. A good IDE will make it very easy to both create and run unit tests as you develop your project.

The most basic form of support for unit tests in an IDE is simply to make it easy to execute arbitrary scripts from within the IDE - so you create a test.pl for your project and keep adding tests to it or to a t/ subdirectory as you develop, and keep running the script as you make changes. All modern IDE's provide at least this minimal capability.

A more sophisticated level of support for unit tests would be the kind of thing that the Java IDE's provide for tests written in JUnit where you can select an existing class file (a .pm file in Perl) and ask the IDE to create a set of stub tests for every subroutine in the file. (See JUnit and the Perl module Test::Unit for more on Unit Tests.) Furthermore, the IDE should support running a set of tests and giving simple visual feedback on what passed/failed. The standard approach in the JUnit world is to show either a "green bar" (all passed) or "red bar" (something failed) and then allow you to see details on failures. Other nice-to-have features include calculating code-coverage, providing statistical summaries of tests, etc.

Here's what a successful run of a Java test suite looks like in Eclipse:

Junit test run, success

Here is the same test run, this time with a failure:

Junit test run, with a failure.

A stack trace of the failure message appears in another part of the window (cropped out here to save space) and if you double-click on the test that failed (testInflate) the IDE will open the file (BalloonTest in this case) and navigate to the test function.

The central idea is that the IDE should make it as painless as possible to add and modify and run tests, so you will do more of it during development.

Language-specific help (click on a keyword and the language-specific help is available)

This is a fairly straightforward idea - the IDE should be able to find and display the appropriate documentation for any keyword in your code, so if you highlight push and ask for help you should see the push entry from the Perl documentation. If you highlight a method or subroutine or other symbol name from an imported module the IDE should display the module's documentation for the selected item. Of course this requires that the documentation be available in a consistent machine-readable form, which is sometimes true, and sometimes not true.

Debugger - run your code under a debugger with real-time display of results

All modern IDE's offer support for running your code under a debugger, usually with visual display of what's going on - state of variables, etc. The Komodo IDE supports debugging Perl that is running either locally or remotely.

Typical support for debugging in an IDE includes the ability to set breakpoints, monitor the state of variables, etc.  Basically the IDE should provide support for all the features of the debugger itself, and in the case of graphical IDE's provide a visual display of what is going on.

Automatic reformatting of code.

This means automatically or on-demand re-indenting and other reformatting of code, for example when you cut-and-paste a chunk of code the IDE should support reformatting the chunk to match the indentation of its new location, or if you change the number of spaces or tabs for each level of indentation, or your convention for the placement of curly-braces then the IDE should support adjusting an entire file or all files in your project.

Seamless handling of multiple languages (e.g. Perl and C for XS modules, or Perl and Java, Perl and PHP, etc.)

Mant large software projects involve multiple languages. This is almost universally true in the case of web applications where the user interface is typically written in HTML, CSS, and JavaScript, and the back-end in one or more of Perl, PHP, Java, Python, Ruby, etc. It is very helpful to have development tools that seamless integrate work done in all the languages, and this is becoming quite common, for example both Komodo and Eclipse support multiple languages.

Automated build and test

This feature can be handled at its most basic level by making it easy to run an aribitrary script from within the IDE and to see its output. This could be as simple as having the IDE configured to have a one-click way of running the traditional Perl module build-and-test commands:

perl Makefile.PL
make
make test

A more advanced version of this feature might involve having the IDE create stub-code to test all of the subroutines in an existing file, and/or to run all the scripts in a specified directory under Test::Harness, or to run a set of tests using Test::Unit::TestRunner or Test::Unit::TkTestRunner (the latter provides a GUI testing framework.)

Conclusion and Recommendations for Improvement

While there are many tools for helping Perl development the current state of the Perl tool pool is still years behind other langauges - perhaps 3-5 years behind when compared to Java tools. While there are several tools for Java that have all the features described above there are virtually none for Perl that have all of them. On the other hand, things are looking up - they are better now than a year ago and a 3-5 gap can be closed in a year or two.

A couple of obvious areas where improvements could be somewhat easy are adding more features to EPIC and Komodo - EPIC is open source so there is potentially a wider pool of talent that could contribute, on the other hand Komodo has a company with money behind it, so people actually get paid to improve it. Hopefully both tools will get better with time.

Another interesting possibility is the development of new IDE's or adding to existings ones by using Adam Kennedy's PPI module, which provides the abililty to parse Perl documents into a reasonable abstract syntax tree, and to manipulate the elements and re-compose the document. There is a new Perl editor project, VIP, that is in the design stages and is intended to be "pluggable" and to have special features to support pair programming.

And finally, I've gathered a couple of lists of links for related material. The first list below consist of IDE's and graphical editors for Perl, and the second list consists of various related articles and web sites. I hope this is all inspirational and helpful.

Current IDE's for Perl

Here is a list of IDE's that support Perl - the list is undoubtably incomplete but should form a good starting point for anyone wishing to look into this further:

Related Topics