Schema generation

From Digital Scholarship Group
Jump to navigation Jump to search

Generating Schemas (and Documentation) from ODD

The ODD Itself

The source customization ODD file for the wwp-store encoding language is in our textbase Subversion repository in textbase/schema/wwp-store.odd. It should be valid against the current or the development version of tei_odds. It should almost be valid against either the current or the development version of tei_customization. (There are a few “errors”, as that schema is designed to help write the ODD, not validate it.) In all 4 of those cases it should also be valid against the associated Schematron rules.

The edition number is stored on /TEI/teiHeader/fileDesc/editionStmt/edition/@n in major.minor.fix format. Any change to the ODD that represents a deliberate change to the set of documents validated by the schema ups the minor version number. Changes to only prose or class names or such may be reflected in only the fix number. The major number is reserved for overhauls or other very significant changes. (The last one, e.g., was the change from the TEI ODD language to the TEI PureODD language — i.e., purging (most of) the RELAX NG from the ODD.) The edition number is usually also reflected on the @n of the <change> element describing the change that brings the ODD up to the specified version.

The Subversion $Id:$ is also stored as the content of that <edition. In general it should not be updated by hand, but rather by Subversion on check-in. (This requires that the svn:keywords property be set, which it is.)

Currently the <schemaSpec> element is stored in <body>; someday soon I would like to move it to <back>, reserving the body for prose discussion.

The ODD file is loosely divided into “sections” delimited by nothing other than box comments. A list of those sections follows, in the order they currently occur in the ODD file. (But since the order of the specification children of <schemaSpec> is not constrained by TEI and is insignificant to an ODD processor, it may have changed in the ODD with no effect.)

Note — the ODD has long since outgrown this format (being a linear sequence of statements divided loosely into sections by comments). It should be properly divided into <specGrp> elements, a re-factoring (which would up the major version number) that I hope to do soon.

element & module inclusion, and element deletion section
<moduleRef>s with @include or @except. If we had any schemaSpec/elementRef elements they would probably be here, too.
ODD/Schematron hack section
section for top-level <constraintSpec>s. In looking at this now, it should either be re-named or divided into two sections (and renamed).
element renaming section
set of <elementSpec> elements for cases in which the primary change being made is to change the name of the element. Often other changes are made, as well, but they are usually “minor” changes. E.g., the <titlePage> element is changed to the <titleBlock> element here, but also has a controlled vocabulary for its @type established here. (Fnote: this is because you cannot have two <elementSpec> elements with the same @ident and @ns attributes in a single ODD, yet.)
element addition section
new elements we have added. Note that this should be a subsection of the EXTENSIONS section, below.
class deletion section
for deletion of entire classes (“scorched earth” approach for attribute deletion; we do not delete any model classes)
attribute deletion section
for removing attributes from elements or classes (the “selective thinning” and “repossession” approaches to attribute removal)
class addition section
for addition of attribute classes. The model classes are added in the “new class subsection” of the “EXPANSIONS” section, not sure why.
other class and macro manipulation section
changes to datatypes, macros, and classes
section for changes to content models of "normal" TEI elements
in some cases the change is only to attributes, not content model
required element section
a complex Schematron check to require certain elements that are optional in TEI
attribute constraint section
constraining some attributes further than vanilla TEI, e.g. constrained value list or required instead of optional
EXPANSIONS
Changes in this section need to be in a non-TEI namespace. Since our entire markup language is in our own non-TEI namespace, this doesn't really mean much to us. But most such changes are segregated to here in order to make changing this to a TEI-conformant customization a bit easier.
class manipulation subsection
non-conformant changes to classes
new class subsection
added model classes
macro addition section
macros we added (to make writing the ODD easier)
stuff leftover from EMPB that I still need to look at
I guess
miscellaneous section
as it says
XInclude section
the XInclude attributes defined in ODD

Generating Closed Schema

Any of the following methods should work. The only one I have thoroughly tested is the first, using the command fiumicino.bash on my system. It is my local front-end to roma, so I don’t have to type a really long command every time.

commandline roma
$ cd /path/to/textbase/schema/
$ /path/to/roma2 --patternprefix=ws_--xsl=/path/to/TEI_Stylesheets_repo --noxsd --nodtd --localsource=/path/to/local/p5.xml ./wwp-store.odd .
Note that this creates both RELAX NG XML syntax and compact syntax output, wwp-store.rng and wwp-store.rnc (unless you forget the ultimate dot, in which case they get stuck in a RomaResults/ directory).
Note that the patternprefix can be whatever you want, but it makes life easier if it matches the value of schemaSpec/@prefix.
Note that localsource is optional. I think the Stylesheets will default to using the file /path/to/TEI_Stylesheets_repo/source/p5subset.xml, but I’m not 100% on that.
Stylesheets “bin” commands
$ cd /path/to/textbase/schema/
$ /path/to/TEI_Stylesheets_repo/bin/teitornc --odd --localsource=/path/to/TEI_Stylesheets_repo/source/p5subset.xml wwp-store.odd
Note: I don’t know why you have to specify localsource, you shouldn’t have to.
If you want XML syntax instead of compact, you can use teitorelax instead of teitornc.
oXygen
Open schema in oXygen.
configure scenario
Only has to be done the first time
Select Document > Transformation > Configure Transformation Scenario(s)…
Select TEI ODD to RELAX NG
Click Save and close (or Apply associated, which does next step, too)
Select Document > Transformation > Apply Transformation Scenario(s)
Output is placed in /path/to/textbase/schema/out/wwp-store.rnc
OxGarage
Surf over to OxGarage
Click on Documents (as “Convert from ?”)
Click on ODD Document
Click on RELAX NG compact syntax
Back towards top of page, click Browse… and then find and select the wwp-store.odd file
Click Convert
Save ZIP file (goes wherever your browser puts it, probably into ~/Downloads/)
Extract the document.rnc file from the ZIP file and rename it
Roma β
Surf over to Roma beta
Should be self-explanatory
Roma
Surf over to Roma
Select Upload a customization, and click Browse… and select the wwp-store.odd file
Click the red Start button
Click the Schema tab
Select RELAX NG schema (compact syntax) from the “Which format do you prefer?” drop-down (it is typically the default)
Click the red Generate button
Save output file (goes wherever your browser puts it, probably into ~/Downloads/)

There are no changes required to the output RELAX NG except (sometimes) moving & renaming the file. However, the compact syntax output is more human-readable if you run it through my “fixer”: /home/syd/bin/fix_rnc_whitespace.perl --patternprefix="ws_" < wwp-store.rnc > FIXED.rnc && mv FIXED.rnc wwp-store.rnc.

Generating Open Schema

There are fewer choices for generating the open schema (i.e., the ISO Schematron). No matter how you initially generate the file, though, it needs to be post-processed. Currently that post-processing is done by hand. Note that oXygen has no built-in process for doing this. (At least, not yet.) Also our ODD just makes Romaβ hang. (I have not told Raff yet. I probably should, but I know he is very busy teaching a new undergraduate course this semester; I believe this is the first time in his career he has taught a course.)

commandline roma
$ cd /path/to/textbase/schema/
$ /path/to/roma2 --patternprefix=ws_--xsl=/path/to/TEI_Stylesheets_repo --noxsd --nodtd --localsource=/path/to/local/p5.xml --isoschematron ./wwp-store.odd .
Note that this creates wwp-store.isosch (unless you forget the ultimate dot, in which case I think it will get put in a RomaResults/ directory).
Note that localsource is optional. I think the Stylesheets will default to using the file /path/to/TEI_Stylesheets_repo/source/p5subset.xml, but I’m not 100% on that.
Stylesheets “bin” commands
$ cd /path/to/Stylesheets_repo/bin/
$ ./teitoschematron --odd /path/to/textbase/schema/wwp-store.odd /path/to/textbase/schema/wwp-store.isosch
OxGarage
Surf over to OxGarage
Click on Documents (as “Convert from ?”)
Click on ODD Document
Click on ISO Schematron constraints
Back towards top of page, click Browse… and then find and select the wwp-store.odd file
Click Convert
Save ZIP file (goes wherever your browser puts it, probably into ~/Downloads/)
Extract the document.rnc file from the ZIP file and rename it
Roma
Exactly the same process as for the closed schema, except select ISO Schematron from the “Which format do you prefer?” drop-down

post-processing

The generated wwp-store.isosch file is NOT usable as is. This is because the Schematron extraction routine does not realize we use an entirely different namespace, and because one of our Schematron processors chokes on multiple declarations of the same namespace. So open up the file in a text editor and perform the following steps.

  • One easy way to get both the <ns> elements and a comment describing what you did right is to copy them from the previous version of the wwp-store.isosch file. (Referred to as “copy-and-replace” below.) You would want to copy from the line immediately following the “This file generated DATETIME by 'extract-isosch.xsl'” to the “constraints” comment, and replace the equivalent portion in the new file. However, if you generate the changes this way a) you still have to perform the s/tei:/wwp:/g; change, and you should definitely test the schema, at least with oXygen if not probatron. This is because once in a while things do change, and a simple copy is insufficient.
  • If you have not used the copy-and-replace method above, look through the <ns> elements near the top of file. (There are generally 6–10 of them divided into up to two sections, “namesapces, declared” and “namespaces, implicit”.)
    • All but one of any set that have the same @prefix but different @uris must be deleted (these are actual Schematron errors). This usually does not occur or occurs only with the one pair, the prefix "tei", which is bound to both the TEI namespace and the WWP namespace. Delete the TEI namespace one. (Keep in mind that "TEI" should remain bound to the TEI namespace.)
    • All but one of any set that have the same @uri but different @prefixs should be deleted (these are not Schematron errors, but probatron treats them as an error when it shouldn’t). This usually does not occur or occurs only with the one pair that bind the WWP namespace to both "tei" and "wwp". Delete the "tei" one.
    • All but one of any set that have the same @prefix and the same @uri should be deleted (these are not Schematron errors, but probatron treats them as an error when it shouldn’t). This usually does not occur.
  • The Schematron was extracted from both the TEI Guidelines and from the customization ODD using paths that address elements in the same WWP namespace. The former use the hard-coded prefix tei:, the latter use wwp:. But (because probatron is lame) we just deleted the declaration for the tei: prefix. So now do a global change, changing all occurences of tei: with wwp:. Be sure not to change TEI: (to either wwp: or WWP:, the latter of which is what many editors would do by default).
  • You may wish to format-and-indent or the equivalent just to make file more human-readable.
  • If you did not use the copy-and-replace method above, make sure to add a comment near top-of-file declaring the changes you made.

Generating HTML Documentation

As with the closed schema, the only generation method below that I have thoroughly tested is the first. As witht he open schema, the output should be tweaked a bit after generation before use.

commandline roma
$ cd /path/to/textbase/schema/
$ /path/to/roma2 --patternprefix=ws_--xsl=/path/to/TEI_Stylesheets_repo --noxsd --nodtd --norelax --dochtml localsource=/path/to/local/p5.xml ./wwp-store.odd .
Note that this creates the desired output in wwp-store.doc.html, and an unused intermediate file wwp-store.doc.xml, both in the current directory (unless you forget the ultimate dot, in which case they get stuck in a RomaResults/ directory).
Note that the patternprefix can be whatever you want, but it makes life easier if it matches the value of schemaSpec/@prefix.
Note that localsource is optional. I think the Stylesheets will default to using the file /path/to/TEI_Stylesheets_repo/source/p5subset.xml, but I’m not 100% on that.
Stylesheets “bin” commands
$ cd /path/to/textbase/schema/
$ /path/to/TEI_Stylesheets_repo/bin/teitohtml --odd --localsource=/path/to/TEI_Stylesheets_repo/source/p5subset.xml wwp-store.odd
Note: I don’t know why you have to specify localsource, you shouldn’t have to.
oXygen
Open schema in oXygen.
configure scenario
Only has to be done the first time
Select Document > Transformation > Configure Transformation Scenario(s)…
Select TEI ODD XHTML
Click Save and close (or Apply associated, which does next step, too)
Select Document > Transformation > Apply Transformation Scenario(s)
Output is (appropriately) placed in /path/to/textbase/schema/wwp-store.html
OxGarage
Surf over to OxGarage
Click on Documents (as “Convert from ?”)
Click on ODD Document
Click on xHTML
Back towards top of page, click Browse… and then find and select the wwp-store.odd file
Click Convert
Save output file (goes wherever your browser puts it, probably into ~/Downloads/)
Roma β
Surf over to Roma beta
Should be self-explanatory
Roma
Surf over to Roma
Select Upload a customization, and click Browse… and select the wwp-store.odd file
Click the red Start button
Click the Documentation tab
Select HTML web page from the “Which format do you prefer?” drop-down (it is typically the default)
Click the red Generate button
Save output file (goes wherever your browser puts it, probably into ~/Downloads/wwp-store_doc.html)

post-processing

  • Rename the output file (which might be wwp-store.doc.html, wwp-store.odd.html, wwp-store_doc.html) to wwp-store.html.
  • If the first line is blank delete it or replace it with “<?xml version="1.0" encoding="UTF-8"?>”. (Web browsers don’t care, but some XML processors consider it an error.)
  • In several places extraneous metadata is output abutted directly onto useful metadata with no intervening space. Conveniently, this almost always occurs in the vicinity of my name.
    • Search for “Bauman” (you can search for the shorter “Syd” instead, but there will be a few more false positives, e.g. due to some examples containing “Sydney Smith”).
    • When my name occurs immediately after the title, delete the name (i.e., change “LanguageSyd Bauman. ” to just “Language”).
    • When my name occurs right before the $Id: Subversion keyword (whether there is intervening whitespace or not) delete everything in between the preceding ‘>’ to the ‘$’ of the “$Id:” (leaving those characters themselves).
  • Be cautious about using format-and-indent (or the equivalent) on the HTML file, as you may introduce undesired whitespace. (Then again, you may not.)

Combined

With some methods, it is possible to generate more than one output at once.

commandline roma
$ cd /path/to/textbase/schema/
$ /path/to/roma2 --patternprefix=ws_--xsl=/path/to/TEI_Stylesheets_repo --noxsd --nodtd --dochtml --isoschematron --localsource=/path/to/local/p5.xml ./wwp-store.odd .
oXygen
When configuring the transformation scenario, select multiple outputs (e.g., TEI ODD XHTML and TEI ODD to RELAX NG (compact syntax))
Roma, Roma β, and OxGarage
You cannot actually generate multiple outputs with one click, but there is no need to re-lead the input ODD file each time you want to generate output.
Retrieved from "https://dsg.northeastern.edu/wiki/index.php?title=Schema_generation&oldid=3680"