I received an e-mail the other day from somebody asking how he could contribute to the development of The State Decoded. As a rule, this is a sign that I’m doing something wrong. In the spirit of addressing that that, here are a list of relatively self-contained, interesting, diverse features that await addition to The State Decoded, that you or somebody you know might be interested in creating, for folks of all levels of technical knowledge and many fields of expertise.

Create the Functionality to Add Laws to a Portfolio

Site users ought to be able to keep track of laws that are of interest of them. Using jQuery’s localstorage.setItem / localStorage.removeItem, provide the functionality to let people add laws to a portfolio, and then create a page where people can see a list of the laws in their portfolio.
Issue #30

Support Memcached and/or ElastiCache

Provide an option in config.inc.php to provide configuration information to connect to the object cache of choice, and modify class.Law.inc.php to cache laws within the cache upon reading them or, if already cached, read them from there, rather than the database. (Presumably laws should be cached in Memcached upon being requested, rather than pre-loading Memcached full of all laws, since most legal codes aren’t liable to fit within a reasonable amount of server memory.)
Issue #263

Establish an Interface for Showing Diffs of a Law

With each new release of a legal code, we add a new edition (tracked in the “editions” table) and add all of those new laws to the “laws” table. Provide the functionality to let somebody look through the various versions of a law over time, or compare two versions to see how they’ve changed.
Issue #363

Add Word, PDF, and EPUB Export

Add new methods to class.ParserController.inc.php to create, at the time that a legal code is imported, Word, PDF, and EPUB versions of the legal code, and portions thereof. (Realistically, this should be three separate issues, since it’s three separate projects.) Ideally there’d be Word and PDF versions of every law, every structural unit (chapter, title, etc.), and the entire legal code, and then EPUB versions of every structural unit and of the entire legal code.
Issue #50

Provide an Option to Use the OpenDyslexic Font

Create a jQuery-based widget to let somebody enable or disable the use of the OpenDyslexic font, by setting a cookie, and then a jQuery-based widget to toggle the use of that typeface for the body font (article#law) if that cookie is set.
Issue #340

Sync Laws to GitHub

Some folks are pretty psyched about putting laws on GitHub, for various reasons. Create a method that will commit the plain text version of all laws in a given edition of the legal code to a specified GitHub repository, and add the necessary options to `config.inc.php` to enable that.
Issue #161

Provide Vagrant Configurations

There’s an effort underway to create a ready-to-go Vagrant configuration of the project, so that it’s trivial for somebody to set up an implementation of The State Decoded on their own system. This sub-project has its own repository, and a couple of issued logged in its own issue tracker.
Issue #284

Display Related Legal Self-Help Documents

I’ve made a first crack at interfacing with ProBonoNet’s API to gather up a list of all of their free self-help legal documents. This needs to be extended, to store this data in a way that’s available to The State Decoded, and then—here’s the hard part—when somebody looks at a law for which there’s a relevant self-help legal document available, we need to be able to identify that document and display text promoting it. We’ve got the UI elements in place for this, but we just lack the glue that allows us to say “this law about foreclosure is probably related to this guide about what to do if your home is being foreclosed on.” Solr may be a good way to make this match.
Issue #162

Edit, Write, or Propose Changes to Documentation

The State Decoded has some pretty decent documentation that’s under active development, but it would strongly benefit from review by people who aren’t contributors to the project. (People who already know a project in great detail aren’t in a great mindset to write about it in a way that beginners can understand.) The documentation is hosted on GitHub, so pull requests can be made directly, or, for folks who aren’t technical, suggestions or proposed changes can be made in the form of an issue report.
Documentation Repository

This isn’t everything that needs to be done, of course—these are just the interesting, relatively self-contained new features. You can see the complete list of outstanding issues on GitHub, or just the list of new features awaiting creation.

At WebLaws.org, Robb Shecter is puzzling through how to deal with the California Code’s curious lack of titles. Most state codes provide a title for each law (known as a “catch line” in most states), such as “Enforcement of child labor law,” “Fees for filing documents or issuing certificates,” or “Money derived from forest reserve.” Not California’s. Robb provides the example of California’s § 459, the law prohibiting burglary. One must read through the law to know what it does. This, of course, makes it very difficult to navigate through the California Code.

The question that Robb asks is what we are to do about this. The problem is abstract for me—I have no immediate prospects of working on the California Code, but Robb has it online now, so the problem is very real for him.

The reason that California has been able to get by with such an odd arrangement is that private legal vendors, like West and LexisNexis, write their own titles for laws. Most attorneys surely use the terminology provided by those companies, some perhaps unaware that those are not official titles. Those titles are copyrighted by those vendors, though, and cannot be used for projects like WebLaws.org or The State Decoded. This means that we must be able to generate our own titles.

Here is the conceptual solution that I arrived at for California some months ago, which I share here in hopes that it might do others some good. I have not implemented this, so while in theory it makes sense, I cannot say for sure that it’ll work.

Like many states, California maintains an annual index of all legislation that has come before their legislature. (This is the 2011–2012 index, for example.) This allows people to look up all bills pertaining, for instance, retirement, and see the following listing:

continuing care retirement communities, AB 748, 1698
pensions—
early distribution penalty waiver, AB 558, 2656
employer-sponsored retirement plans, SB 1234
golden state retirement savings trust, SB 1234
rollover funds, tax-free: medical and long-term care premiums, SJR 21
secure choice retirement savings trust, california, SB 1234
public retirement systems. See name of particular retirement system (e.g., PUBLIC EMPLOYEES’ RETIREMENT SYSTEM).
unemployment compensation benefits, AB 2310

This looks to me like a rich source of titles.

The process is straightforward. First, match up all legislation with the existing law that it proposes to amend. Then, find every entry for all of that legislation in the index of legislation. The description in the index becomes the title of the law. For those laws that have multiple candidate descriptions (either because they’re in the index repeatedly, multiple bills propose to amend them in a given year, or there are many years of attempted amendments), the words that appear most frequently in those descriptions can be used to automatically assemble a title.

This is bound to lead to some goofy titles. And some laws have not had bills introduced that would amend them for decades, and so information about them would not be available in bulk. But in my experience, the laws that most interest people are the ones that legislators attempt to amend, so titles would be provided for those laws that are most liable to be read.

What of the rest of the laws, left untitled by this first method? Statistically improbable phrases (SIPs) are a good backup method. A phrase that occurs in a law that is very rarely found in the rest of California’s laws is liable to be a decent candidate for its title. Again, potentially goofy titles could result, and I have not tested this, but theoretically it could work pretty well. Amazon.com displays SIPs for some of their books, and I think those illustrate the range of results that one could expect from them. For instance, Nora Ephron’s newly re-popular “I Feel Bad About My Neck” has two SIPs: “serial monogamy,” “cabbage strudel.” The former is a not-unreasonable summation of of the book. The latter is obviously pretty unreasonable.

Some experimentation is going to be necessary to arrive at a decent system for generating titles for California’s laws. Ideally, whomever creates them would put them up on Google Docs for some collaborative editing, and release the resulting text under an open license, so that, at last, we will all have titles for all of the laws in the California Code.

This problem is, not incidentally, emblematic of a routine problem with state codes. It seems like they’re all missing something, some core piece of data that would make them far more useful. Each of these will require its own patch, its own work-around, to render those laws widely accessible to the general public. We’re all taking it one state at a time.

There is a common belief that since laws are the result of legislation, then surely one can automatically assemble an amended version of the code based on the bills that have passed the legislature. This is both a really cool idea and wrong.

Your standard narrative of how a bill becomes law doesn’t really cover what it purports to cover. Usually what’s really being explained is how a bill passes, but not how it becomes law. There’s a whole process between the passage of a bill and the encoding of that bill in a state’s codified laws.

Legislatures pass hundreds or thousands of bills every year, some of which are budgetary, some of which are to define their own rules, some of which pertain to state administration, and some of which are resolutions. The remainder are patches to be applied to law, either proposing a new section or amending an existing one (by either adding or removing material). These patches look familiar to anybody who has seen a prettied-up diff, and it’s wholly logical to figure that amending the state’s laws should simply mean collecting all of the bills that pass and applying those patches to the laws.

The trouble is that these patches are not always the last word on the changes that are going to be made to existing law.

Virginia is the state in which I have the most knowledge of this process, so I’ll provide a few words about my state’s process. The Virginia Code Commission is a tiny state agency, with just seven employees, who is charged with overseeing the code. Their duties are spelled out in Title 30 (General Assembly), Chapter 15 (Virginia Code Commission) of the state code, but the interesting bit is § 30-149 (Authority for minor changes to the Code of Virginia):

The Commission may correct unmistakable printer’s errors, misspellings and other unmistakable errors in the statutes as incorporated into the Code of Virginia, and may make consequential changes in the titles of officers and agencies, and other purely consequential changes made necessary by the use in the statutes of titles, terminology and references, or other language no longer appropriate.

The Commission may renumber, rename, and rearrange any Code of Virginia titles, chapters, articles, and sections in the statutes adopted, and make corresponding changes in lists of chapter, article, and section headings, catchlines, and tables, when, in the judgment of the Commission, it is necessary because of any disturbance or interruption of orderly or consecutive arrangement.

The Commission may correct unmistakable errors in cross-references to Code of Virginia sections and may change cross-references to Code of Virginia sections which have become outdated or incorrect due to subsequent amendment to, revision, or repeal of the sections to which reference is made.

The Commission may omit from the statutes incorporated into the Code of Virginia provisions which, in the judgment of the Commission, are inappropriate in a code, such as emergency clauses, clauses providing for specific nonrecurring appropriations and general repealing clauses.

(TL;DR: The Code Commission can make a lot of changes during that in-between period when a bill has passed, but it’s not quite yet a law.)

It is not unusual for the legislature to amend a bill at the very last minute, without proper review. From the marked-up format of a bill (words crossed out, others inserted) what can emerge is grammatically incorrect or even contains logical errors. It’s surely a judgment call whether such problems can be fixed by the Virginia Code Commission or whether it will require the General Assembly to fix them, which may well require a delay of nearly a year.

Here, for instance, is a selection of the 35 changes made to the Code of Virginia by Code Commission staff since the 2011 edition was published:

In Virginia—as in other states, although I don’t know how many—an attempt to use legislation as a changelog for the state code would yield results that would be very convincing-looking, but that would deviate substantially from the official code. Bills must frequently pass through a human filter before they become laws. That’s not something that you can simulate with a Ruby gem or a PEAR package. Some things just require some thought, in ways that can’t yet be automated.

Those of us who work with big data have a tendency to describe working with it in cavalier terms. “Oh, I just grabbed the XML file, wrote a quick parser to turn it into CSV, bulk loaded it into MySQL, laid an API on top of it, and I was done.” The truth is that things very rarely go so well.

Real-world data is messy. Data doesn’t convert correctly the first time (or, often, the tenth time.) File formats are invalid. The provided data turns out to be incomplete. Parser code that was so straightforward when written for the abstract concept of this data quickly turns into a series of conditionals to deal with all of the oddities of the real data.

While I’ve been developing the parser for The State Decoded, it’s become obvious that state laws themselves are too messy to standardize entirely, and the data formats in which states provide those laws are, in turn, too messy to import easily.

As a case study of the messiness of real-world data, here are some of the challenges I’ve encountered in parsing state legal codes.
 

Encoding Errors

A precious few states provide their state codes as bulk data. While it might seem like a real gift to get an XML file of every state law, if the XML is invalid, then that’s really more of a white elephant. One state provided me with SGML that they had, in turn, been provided with by LexisNexis. It was riddled with hundreds of errors, and could not be parsed automatically. After hours of attempting to fix problems by hand, I finally threw in the towel. Weeks later, LexisNexis provided a corrected version, and my work could continue.

Often, working with big data means doing pathbreaking work. The result is that sometimes nobody has ever before attempted to do anything with the data sets in question. Assembled by well-meaning but inexperienced people, those data sets may consequently be encoded incorrectly.

Changing Realities

State laws are occasionally restructured, renumbering huge portions of the code. Virginia’s entire criminal code—Title 18.1— became Title 18.2 about fifteen years ago. No redirects exist in 18.1, no pointers to the new location, no sign of what was. One must simply know that it changed. Court cases, articles from legal journals, or attorney generals’ opinions that cited sections of code within 18.1 are thus either useless or must be passed through a hand-crafted filter to point the citation to the new section number.

It would be nice if reality would consent to remain static to ease the process of cataloging it. But the world changes, and data reflects those changes. That can make it awfully frustrating to parse and apply that data, but that’s just the price of admission.

Inconsistencies in the Data

There are at least a few states who violate their standard state code structure. They might structure their code by dividing it into titles, each title into chapters, and each chapter into sections. Except, sometimes, when chapters are called “articles.” Why do they do this? I have no idea. If lawmakers consulted with database developers prior to recodifying their state’s laws, no doubt our legal codes would be normalized properly.

These inconsistencies might be illogical, but they’re how it is, and must be reflected in the final application of the data. This can be particularly frustrating if the provided bulk data doesn’t record these inconsistencies internally, requiring the gathering of external information to be applied to the data, as is often the case.

Missing Data

One state’s code contains periodic parenthetical asides like “See Editor’s note.” What does the editor’s note say? There’s no way to tell—it’s not part of the bulk data or the state’s official website for their legal code. Those editor’s notes will have to be obtained from the state’s code commission, which will probably come in the form of a bunch of Word files attached to an e-mail.

Not all data exists electronically, and not all data that exists electronically exists in a single location. Often, piecing together a meaningful data set requires gathering information from disparate sources, sometimes in awkward ways. And sometimes the last few bits of data just aren’t available, and the data set is going to have to be incomplete.
 

All of these problems are solvable, in one way or another, but those solutions can be time-consuming. The ratio of the Pareto principle applies here: one is liable to get 80% of the data set whipped into shape in the first 20% of the time. The remaining 20% of the data will require the remaining 80% of the time. That first 80% feels magical—everything just falling into place—but that last 20% is just plain hard work.

Real-world data is messy. Working with big data means cleaning it up.