WWO IP Updating

From Digital Scholarship Group
Jump to navigation Jump to search

Updating subscriber password and IPrange files

NEW Summary aimed at Sarah Connell

  • Export password file to passexport[date].tab, where [date] is whatever you want to use to keep the different files separate in your head.
  • Export IP address file to ipexport[date].tab, where [date] is whatever you want to use to keep the different files separate in your head.
  • Test the IP address file by running check_IPs.perl < /path/to/ipexport[date].tab; fix any errors.
  • Instead of, or in addition to, putting those files in ~/Dropbox/WWPshare/, upload them to wwp-test by issuing scp ipexport[date].tab syssa.connell@wwp-test.neu.edu:/tmp/ipexport.new and scp passexport[date].tab syssa.connell@wwp-test.neu.edu:/tmp/passexport.new from the commandline (i.e. in Terminal).
  • Log into wwp-test by issuing ssh syssa.connell@wwp-test.neu.edu.
  • Issue cd /etc/httpd/passwd/ to change your current working directory to the right place
  • Issue the command ./update_IPs_and_PWs.bash, which will do most the work; note that you will probably have to re-enter your password for wwp-test in order to re-start the web server
  • If everything works (keep your fingers crossed), check in with something like svn ci -m "Added new users from Starfleet Academy for testing on space stations"
  • If everything works, push update from wwp-test to wwp by issuing /etc/httpd/passwd/sync_passwd
  • Log off of wwp-test by typing ^d or by entering logout

OLD Summary aimed at Sarah Connell

IP file

  • Export IP address file to ipexport[date].tab, where [date] is whatever you want to use to keep the different files separate in your head.
  • Instead of, or in addition to, putting file in ~/Dropbox/WWPshare/, upload it to wwp-test by issuing scp ipexport[date].tab saraheconnell@wwp-test.neu.edu: from the commandline (i.e. in Terminal).
  • Log into wwp-test by issuing ssh saraheconnell@wwp-test.neu.edu.
  • Fix any line-end issues by issuing perl -p -i -e 's/\r/\n/g;' ipexport[date].tab
  • Issue cp ipexport[date].tab /etc/httpd/passwd/textbase_ipaddrs
  • Issue sudo service httpd --full-restart
  • Check in with something like svn ci -m "Added IP range for Hogwarts"
  • Push update from wwp-test to wwp by issuing /etc/httpd/passwd/sync_passwd
  • Log off of wwp-test by typing ^d or by entering logout

Password file

  • Export password file to passexport[date].tab, where [date] is whatever you want to use to keep the different files separate in your head.
  • Instead of, or in addition to, putting file in ~/Dropbox/WWPshare/, upload it to wwp-test by issuing scp passexport[date].tab saraheconnell@wwp-test.neu.edu: from the commandline (i.e. in Terminal).
  • Log into wwp-test by issuing ssh saraheconnell@wwp-test.neu.edu.
  • Issue cd /etc/httpd/passwd/
  • Issue ./add_passwds textbase_htpasswd textbase_htgroup < ~/passexport[date].tab
  • Issue sudo service httpd --full-restart
  • Check in with something like svn ci -m "Added new users from Starfleet Academy for testing on space stations"
  • Push update from wwp-test to wwp by issuing ./sync_passwd
  • Log off of wwp-test by typing ^d or by entering logout

Both

  • Export both ipexport[date].tab and passexport[date].tab, as above
  • Upload both by issueing scp ipexport[date].tab passexport[date].tab saraheconnell@wwp-test.neu.edu:
  • Log into wwp-test by issuing ssh saraheconnell@wwp-test.neu.edu.
  • Issue perl -p -i -e 's/\r/\n/g;' ipexport[date].tab
  • Issue cd /etc/httpd/passwd/
  • Issue cp ~/ipexport[date].tab textbase_ipaddrs
  • Issue ./add_passwds textbase_htpasswd textbase_htgroup < ~/passexport[date].tab
  • Issue sudo service httpd --full-restart
  • Check in with something like svn ci -m "Changed IP range for Hudson University; added new users Moe, Larry, and Curly"
  • Push update from wwp-test to wwp by issuing ./sync_passwd
  • Log off of wwp-test by typing ^d or by entering logout

Instructions aimed at Syd

See on wwp-test:

/etc/httpd/passwd/README-how_to_update_ipaddrs

For convenience, the instructions are copied below — but note that Syd has updated the instructions here on the Wiki, but not in that file (yet).

How to update the IP and password authentication files for the WWO

Perform all these operations on wwp-test. Then use the sync process to push them to wwp.

Simple summary

  • export IP addresses and passwords
  • fix to Unix line ends and run “Fix IP export”
  • upload to wwp-test
  • cd /etc/httpd/passwd/
  • Update the passwords: ./add_passwds textbase_htpasswd textbase_htgroup < new_passexport_file
  • Copy ip export file to the test position: cp new_ipexport_file textbase_ipaddrs.test
  • Restart web server: sudo /usr/sbin/apachectl graceful
  • Load the test page (http://www.wwp-test.neu.edu/manual/) in a browser
  • ⁈ Check error log: less /var/log/apache2/wwptest_error_log (type > to go to end of the file.)
  • Copy the test file to the active position: cp textbase_ipaddrs.test textbase_ipaddrs
  • Restart web server again: sudo /usr/sbin/apachectl graceful
  • Check-in changes: svn ci -m "message"
  • Synchronize wwp with wwp-test: ./sync_passwd

Extended instructions

The first step is to export the IP addresses and passwords from the two FileMaker databases that hold this information, the WWP_DeliveryManagement and WWP_DeliveryIndividuals databases. In the WWP_DeliveryManagement database, go to the “Export IP addresses” layout and then select the “Find for export button” at the top of the layout. This will filter the records so that only those which contain active licenses and IP information will appear on the list. Then select the “Export IP addresses” button. In the Save dialog, make sure that the “Field export order” box contains the following fields, in this order:

  • IP Addresses
  • #
  • techEmail
  • Institution Name

Make sure that you have selected the export format to be tab-delimited text. Once you click “Export”, the file will be saved with whatever name you have chosen to give it (generally something like ipexport_2014-04-01.tab).

In the WWP_DeliveryIndividuals database, click the “Export passwords” button that appears on the Overview layout screen. This will automatically take you to the Export dialog. Choose a name for the exported file (generally something like passexport_2014-04-01.tab), make sure that the export format is tab-delimited text, then select “Save”. In the “Field export order” box, make sure that the following fields appear in this order:

  • logon
  • password

Fix any errors which Filemaker introduces into the tab-delimited ipexport_someDate.tab file. As of the writing of this README file, there are now no longer any such errors, except for the ^K problem described below, but new ones might crop up (hope not).

Remember, there should be no linebreaks in any given line. Each school’s information occupies exactly one line. Also make sure BBEdit saved with Unix line-ends, because Mac line-ends will mess everything up. If you do a less and see what looks like a ^M then you have Mac line ends. You can either use your favorite editor (BBEdit does this nicely; use C-x RET f in Emacs) or use this Perl one-liner to change them to Unix ones: 

perl -pi -e 's/\r/\n/g;' [filename(s)]

E.g., to fix all files with “export” in the name: 

perl -pi -e 's/\r/\n/g;' *export*

Note also that it seems that FileMaker 9 exports ^K characters (that’s the U+000B character, which is typed “control-k”) instead of spaces (U+0020 characters) to separate items within a field. The process below expects spaces; luckily ^K does not occur anywhere else, so all you need to do is change all occurences of ^K to space.

To do this in TextWrangler:

  • open the file
  • make sure the insertion point is before the 1st character in the file (cmd-up, cmd-left should do the trick)
  • open the find/replace dialog box (either cmd-F or choose “Find...” from the “Search” menu)
  • if you have a preset pattern: select it from the “Patterns:” pop-up menu which is just above the left side of the search-for text-box
  • if not:
    • enter a single ^K character into the search-for text box; you can do this either by typing “\x0B”, or, if you switch to Unicode Hex Input, by typing “opt-0 opt-0 opt-0 opt-B” (because it is character U+000B)
    • enter a single space into the replace-with text-box
  • click on “Replace All”

Alternatively, you can change them via a Perl one-liner in a shell: 

perl -pi -e 's/^K/ /g;' [filename(s)]

E.g., to do both of these changes at once on all files with “export” in the name: 

perl -pi -e 's/\r/\n/g; s/^K/ /g;' *export*


  1. Copy the files to someplace on wwp-test (e.g., your home directory or the /tmp/ directory) using scp (or any SFTP client of your choosing) and then ssh into wwp-test.
  2. cd /etc/httpd/passwd/
  3. check that content is up-to-date: svn stat -u
  4. update if needed (i.e., there are “*” files, very unlikely): svn up
  5. For the Password file, run this command:
./add_passwds textbase_htpasswd textbase_htgroup < [/path/to/new/password/export/file]

E.g., if you put passexport_2014-08-25.tab into your home directory: 

./add_passwds textbase_htpasswd textbase_htgroup < ~/IPs/passexport_2014-08-25.tab

If you get an error message like “Too many fields, stdin, line 1 at ./add_passwds line 69, <STDIN> line 1.”, one likely reason is that you have forgotten to convert Macintosh or DOS linends to Unix linends (see above). For more information on this command see the README.goerwitz.passwords your current directory (by Richard Goerwitz). Don’t worry about the warning messages concerning the format of the userids, unless one of the userids has an apostrophe in it. For example, Susan_O'Hara was problematic, so I had Julia change it to Susan_OHara.

  1. For the IP address file:

Copy the IP export file into your current directory (/etc/httpd/passwd/), overwriting the file textbase_ipaddr.test, e.g.: 

cp /tmp/ipexport_2014-08-25.tab textbase_ipaddrs.test
  1. Restart the web server:
sudo /usr/sbin/apachectl graceful
  1. Find out if there are any errors by loading the test page (http://www.wwp-test.neu.edu/manual/) in a browser. (Wait about 15-20 seconds before doing this, since hitting the server before that interval sometimes produces no result.)
  2. Any errors generated will be found in the file:
/var/log/httpd/error_log

Find the line in the log that looks “Graceful restart requested, doing restart” followed a few lines later by “Apache/2.2.3 (Red Hat) configured -- resuming normal operations” and then a series of lines that begin “proxy_util.c”. That's what I will refer to below as the server’s start/stop block of messages (a phrase I just made up).

After that, if there are a bunch of lines that end in “OK”, you should be all set, e.g.: 

[Wed Mar 15 17:17:04 2006] [debug] IPAccess.pm(157): Apache2::IPAccess:  Constructing IP table from /etc/httpd/passwd/textbase_ipaddrs.test.
[Wed Mar 15 17:17:06 2006] [debug] IPAccess.pm(329): Apache2::IPAccess:  Remote address, 128.148.157.101, matches
 16-bit mask in the IP table.  OK!
 [etc.]

If other people happen to be hitting teller at the same time you did your test, other people’s IP addresses (and other error or access messages) may be mixed in with yours, or near yours, so it’s important to check the log as soon as you can after loading the test manual/ page, so that your hits will be right there after the server’s start/stop block of messages. This just makes it easier to find. As you can see, you will find your own IP address in those lines (above, you can see mine is 128.148.157.101). That and the timestamp also make it easier to eyeball.

If there is a problem with the syntax of the ipaddrs file, this will be indicated by almost anything else (sadly I cannot enumerate all the possible problems), e.g.: 

[Mon Sep 29 20:17:24 2003] [warn] Apache::IPAccess:  Address, 128.196.252.181-199, in /www/passwd/textbase_ipaddrs, line 81 covers a subset  of 128.196.229-255.0, at line 25.

At the time this error occurred, I sent this email to Julia: 

--------------- email ----------------------------------------------
	There is an overlap in the IPs of these two universities:
		University of Arizona:
		128.196.77-227.0/255.255.255.0

		Northern Arizona University:
		128.196.252.181-199/255.255.255.255

	You will need to find out from them which one really owns that range. The
	previous export said it belonged to Northern Arizona, so maybe UofA is
	incorrect in saying it is theirs?
    --------------- /email ----------------------------------------------

Now, that is the last major error we had, and it happened over 2 years ago, so normally I don’t think you will have any problems!

  1. FIX the errors (this could involve emailing Julia and then having her email the institutions/librarians in question, to resolve the matter, which is why you have to use this 'test' file 'textbase_ipaddrs.test' instead of the real file!), then save the file and restart the server again, and so on, until all errors are gone.
  1. Copy the test file to the real file, overwriting the real file:
cp textbase_ipaddrs.test textbase_ipaddrs

And then restart apache again.

  1. Run the script that synchronizes wwp-test's /etc/httpd/passwd/ directory with that on wwp:
./sync_passwd

It prompts for the current user’s password on wwp; just enter the password and the script runs very quickly. Note that it syncronizes the entire /etc/httpd/passwd/ tree via rsync. Thus there is no separate command to sync the IP addresses apart from the passwords.

There should be no need to restart the Apache on wwp, although if there's a problem with access on wwp following a sync, that should be the first debugging step.

That’s it - updating process complete.

Note: If this process runs smoothly, but the next day you get a message from the cron daemon that runs the stats collection scripts, the chances are you have an extra whitespace somewhere in the IP range(s) for the institution it mentions. Delete the extra space and run the process again. Also be sure to correct the Filemaker record that’s causing the problem.

OR

The problem could also be caused by permissions conflicts (for instance, when the owner of the cron job is not the same as the owner of the stats collection directories (generally /var/www/html/WWO/awstats/ and /var/www/html/WWO/awstatsMain). In these cases, you either need to have the cron job run by the same user who owns those directories (and the files in them) or change the ownership for everything in those directories so it’s the same as the ownership of cron job.

Retrieved from "https://dsg.northeastern.edu/wiki/index.php?title=WWO_IP_Updating&oldid=3845"