Penguin
Recent Changes
Blame: ncftpspooler(1)
EditPageHistoryDiffInfoLikePages
Annotated edit history of ncftpspooler(1) version 1, including all changes. View license author blame.
Rev Author # Line
1 perry 1 ncftpspooler
2 !!!ncftpspooler
3 NAME
4 SYNOPSIS
5 OPTIONS
6 DESCRIPTION
7 JOB FILE NAMES
8 JOB FILE CONTENTS
9 DIAGNOSTICS
10 BUGS
11 AUTHOR
12 SEE ALSO
13 ----
14 !!NAME
15
16
17 ncftpspooler - Global batch FTP job processor daemon
18 !!SYNOPSIS
19
20
21 __ncftpspooler__ -d [[''options'']
22
23
24 __ncftpspooler__ -l [[''options'']
25 !!OPTIONS
26
27
28 __Command line flags:__
29
30
31 __-d__
32
33
34 Begin background processing of FTP jobs in the designated
35 FTP job queue directory.
36
37
38 __-q__ ''XX''
39
40
41 Use this option to specify a directory to use as the FTP job
42 queue instead of the default directory,
43 /var/spool/ncftp.
44
45
46 __-o__ ''XX''
47
48
49 Use this option to specify a filename to use as the log
50 file. By default, (and rather inappropriately) the program
51 simply uses a file called ''log'' in the job queue
52 directory. If you don't want a log, use this option to
53 specify /dev/null.
54
55
56 __-l__
57
58
59 Lists the contents of the job queue directory.
60
61
62 __-s__ ''XX''
63
64
65 When the job queue is empty, the program sleeps 120 seconds
66 and then checks again to see if a new job has been
67 submitted. Use this option to change the number of seconds
68 used for this delay.
69 !!DESCRIPTION
70
71
72 The ''ncftpspooler'' program evolved from the
73 ''ncftpbatch'' program. The ''ncftpbatch'' program was
74 originally designed as a ``personal FTP spooler'' which
75 would process a single background job a particular user and
76 exit when it finished; the ''ncftpspooler'' program is a
77 ``global FTP spooler'' which stays running and processes
78 background jobs as they are submitted.
79
80
81 The job queue directory is monitored for specially-named and
82 formatted text files. Each file serves as a single FTP job.
83 The name of the job file contains the type of FTP job
84 (''get'' or ''put''), a timestamp indicating the
85 earliest the job should be processed, and optionally some
86 additional information to make it easier to create unique
87 job files (i.e. a sequence number). The contents of the job
88 files have information such as the remote server machine to
89 FTP to, username, password, remote pathname,
90 etc.
91
92
93 Your job queue directory must be readable and writable by
94 the user that you plan to run ''ncftpspooler'' as, so
95 that jobs can be removed or renamed within the
96 queue.
97
98
99 More importantly, the user that is running the program will
100 need adequate privileges to access the local files that are
101 involved in the FTPing. I.e., if your spooler is going to be
102 processing jobs which upload files to remote servers, then
103 the user will need read permission on the local files that
104 will be uploaded (and directory access permission the parent
105 directories). Likewise, if your spooler is going to be
106 processing jobs which download files, then the user would
107 need to be able to write to the local
108 directories.
109
110
111 Once you have created your spool directory with appropriate
112 permissions and ownerships, you can run ''ncftpspooler
113 -d'' to launch the spooler daemon. You can run additional
114 spoolers if you want to process more than FTP job from the
115 same job queue directory simultaneously. You can then
116 monitor the log file (i.e., using ''tail -f'' ) to track
117 the progress of the spooler. Most of the time it won't be
118 doing anything, unless job files have appeared in the job
119 queue directory.
120 !!JOB FILE NAMES
121
122
123 When the ''ncftpspooler'' program monitors the job queue
124 directory, it ignores any files that do not follow the
125 naming convention for job files. The job files must be
126 prefixed in the format of ''X-YYYYMMDD-hhmmss'' where
127 ''X'' denotes a job type, ''YYYY'' is the four-digit
128 year, ''MM'' is the two-digit month number, ''DD'' is
129 the two-digit day of the month, ''hh'' is the two-digit
130 hour of the day (00-23), ''mm'' is the two-digit minute,
131 and ''ss'' is the two-digit second. The date and time
132 represent the earliest time you want the job to be
133 run.
134
135
136 The job type can be ''g'' for a get (download from remote
137 host), or ''p'' for aput (upload to remote
138 host).
139
140
141 As an example, if you wanted to schedule an upload to occur
142 at 11:45 PM on December 7, 2001, a job file could be
143 named
144
145
146 p-20011207-234500
147
148
149 In practice, the job files include additional information
150 such as a sequence number or process ID. This makes it
151 easier to create unique job file names. Here is the same
152 example, with a process ID and a sequence
153 number:
154
155
156 p-20011207-234500-1234-2
157
158
159 When submitting job files to the queue directory, be sure to
160 use a dash character after the ''hhmmss'' field if you
161 choose to append any additional data to the job file
162 name.
163 !!JOB FILE CONTENTS
164
165
166 Job files are ordinary text files, so that they can be
167 created by hand. Each line of the file is a key-pair in the
168 format ''variable''=''value'', or is a comment line
169 beginning with an octothorpe character (''#''), or is a
170 blank line. Here is an example job file:
171
172
173 # This is a NcFTP spool file entry.
174 job-name=g-20011016-100656-008299-1
175 op=get
176 hostname=ftp.freebsd.org
177 xtype=I
178 passive=1
179 remote-dir=pub/FreeBSD
180 local-dir=/tmp
181 remote-file=README.TXT
182 local-file=readme.txt
183
184
185 Job files are flexible since they follow an easy-to-use
186 format and do not have many requirements, but there are a
187 few mandatory parameters that must appear for the spooler to
188 be able to process the job.
189
190
191 ''op''
192
193
194 The operation (job type) to perform. Valid values are
195 ''get'' and ''put''.
196
197
198 ''hostname''
199
200
201 The remote host to FTP to. This may be an IP address or a
202 DNS name (i.e. ftp.example.com).
203
204
205 For a regular ''get'' job, these parameters are
206 required:
207
208
209 ''remote-file''
210
211
212 The pathname of the file to download from the remote
213 server.
214
215
216 ''local-file''
217
218
219 The pathname to use on the local server for the downloaded
220 file.
221
222
223 For a regular ''put'' job, these parameters are
224 required:
225
226
227 ''local-file''
228
229
230 The pathname of the file to upload to the remote
231 server.
232
233
234 ''remote-file''
235
236
237 The pathname to use on the remote server for the uploaded
238 file.
239
240
241 For a recursive ''get'' job, these parameters are
242 required:
243
244
245 ''remote-file''
246
247
248 The pathname of the file or directory to download from the
249 remote server.
250
251
252 ''local-dir''
253
254
255 The directory pathname to use on the local server to contain
256 the downloaded items.
257
258
259 For a recursive ''put'' job, these parameters are
260 required:
261
262
263 ''local-file''
264
265
266 The pathname of the file or directory to upload to the
267 remote server.
268
269
270 ''remote-dir''
271
272
273 The directory pathname to use on the remote server to
274 contain the uploaded items.
275
276
277 The rest of the parameters are optional. The spooler will
278 attempt to use reasonable defaults for these parameters if
279 necessary.
280
281
282 ''user''
283
284
285 The username to use to login to the remote server. Defaults
286 to ``anonymous'' for guest access.
287
288
289 ''pass''
290
291
292 The password to use in conjunction with the username to
293 login to the remote server.
294
295
296 ''acct''
297
298
299 The account to use in conjunction with the username to login
300 to the remote server. The need to specify this parameter is
301 extremely rare.
302
303
304 ''port''
305
306
307 The port number to use in conjunction with the remote
308 hostname to connect to the remote server. Defaults to the
309 standard FTP port number, 21.
310
311
312 ''host-ip''
313
314
315 The IP address to use in conjunction with the remote
316 hostname to connect to the remote server. This parameter can
317 be used in place of the ''hostname'' parameter, but one
318 or the other must be used. This parameter is commonly
319 included along with the ''hostname'' parameter as
320 supplemental information.
321
322
323 ''xtype''
324
325
326 The transfer type to use. Defaults to binary transfer type
327 (TYPE I). Valid values are ''I'' for binary, ''A'' for
328 ASCII text.
329
330
331 ''passive''
332
333
334 Whether to use FTP passive data connections (PASV) or FTP
335 active data connections (PORT). Valid values are ''0''
336 for active, ''1'' for passive, or ''2'' to try
337 passive, then fallback to active. The default is
338 ''2''.
339
340
341 ''recursive''
342
343
344 This can be used to transfer entire directory trees. By
345 default, only a single file is transferred. Valid values are
346 ''yes'' or ''no''.
347
348
349 ''delete''
350
351
352 This can be used to delete the source file on the source
353 machine after successfully transferring the file to the
354 destination machine. By default, source files are not
355 deleted. Valid values are ''yes'' or
356 ''no''.
357
358
359 ''job-name''
360
361
362 This isn't used by the program, but can be used by an entity
363 which is automatically generating job files. As an example,
364 when using the ''-bbb'' flag with ''ncftpput'', it
365 creates a job file on stdout with a ''job-name''
366 parameter so you can easily copy the file to the job queue
367 directory with the suggested job name as the job file
368 name.
369
370
371 ''pre-ftp-command''
372
373
374 ''post-ftp-command''
375
376
377 These parameters correspond to the ''-W'', and ''-Y''
378 options of ''ncftpget'' and ''ncftpput''. It is
379 important to note that these refer to RFC959 File Transfer
380 Protocol commands and __not__ shell commands, nor
381 commands used from within /usr/bin/ftp or
382 ncftp.
383
384
385 ''pre-shell-command''
386
387
388 ''post-shell-command''
389
390
391 These parameters provide hooks so you can run a custom
392 program when an item is processed by the spooler. Valid
393 values are pathnames to scripts or executable programs. Note
394 that the value must not contain any command-line arguments
395 -- if you want to do that, create a shell script and have it
396 run your program with the command-line arguments it
397 requires.
398
399
400 Generally speaking, ''post-shell-command'' is much more
401 useful than ''pre-shell-command'' since if you need to
402 use these options you're more likely to want to do something
403 after the FTP transfer has completed rather than before. For
404 example, you might want to run a shell script which pages an
405 administrator to notify her that her 37 gigabyte file
406 download has completed.
407
408
409 When your custom program is run, it receives on standard
410 input the contents of the job file (i.e. several lines of
411 ''variable''=''value'' key-pairs), as well as
412 additional data the spooler may provide, such as a
413 ''result'' key-pair with a textual description of the
414 job's completion status.
415
416
417 ''post-shell-command'' update a log file named
418 /var/log/ncftp_spooler.
419
420
421 #!/usr/bin/perl -w
422
423
424 my ($line);
425 my (%params) = ();
426
427
428 while (defined($line =
429 $params{$1} = $2
430 if ($line =~ /^([[^=#s]+)=(.*)/);
431 }
432
433
434 if ((defined($params{
435 ($params{
436 {
437 open(LOG,
438 or exit(1);
439 print LOG
440 print LOG
441 print LOG
442 close(LOG);
443 }
444 !!DIAGNOSTICS
445
446
447 The log file should be examined to determine if any
448 ''ncftpspooler'' processes are actively working on jobs.
449 The log contains copious amounts of useful information,
450 including the entire FTP control connection conversation
451 between the FTP client and server.
452 !!BUGS
453
454
455 The ''recursive'' option may not be reliable since
456 ''ncftpspooler'' depends on functionality which may or
457 may not be present in the remote server software.
458 Additionally, even if the functionality is available,
459 ''ncftpspooler'' may need to use heuristics which cannot
460 be considered 100% accurate. Therefore it is best to create
461 individual jobs for each file in the directory tree, rather
462 than a single recursive directory job.
463
464
465 For resumption of downloads to work, the remote server must
466 support the FTP ''SIZE'' and ''MDTM'' primitives. Most
467 modern FTP server software can do this, but there are still
468 a number of bare-bones ''ftpd'' implementations which do
469 not. In these cases, ''ncftpspooler'' will re-download
470 the file in entirety each time until the download
471 succeeds.
472
473
474 The program needs to be improved to detect jobs that have no
475 chance of ever completing successfully. There are still a
476 number of cases where jobs can get spooled but get retried
477 over and over again until a vigilant sysadmin manually
478 removes the jobs.
479
480
481 The spool files may contain usernames and passwords stored
482 in cleartext. These files should not be readable by any user
483 except the user running the program!
484 !!AUTHOR
485
486
487 Mike Gleason, NcFTP Software
488 (mgleason@ncftp.com).
489 !!SEE ALSO
490
491
492 ''ncftpbatch(1), ncftp''(1), ncftpput(1),
493 ncftpget(1), uucp(1).
494 ----
This page is a man page (or other imported legacy content). We are unable to automatically determine the license status of this page.