forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
6492 lines (4732 loc) · 223 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v6r20-pre4]
*Core
CHANGE: Removed deprecated option "ExtraModules" (dirac-configure, dirac-install scripts)
*Framework
CHANGE: ComponentInstaller - remove the old web portal configuration data
used during the installation
*WMS
NEW: PilotLoggingDB, Service and Client for handling extended pilot logging
NEW: added a new synchronizer for Pilot3: sync of subset of CS info to JSON file,
and sync of pilot3 files
*DMS
NEW: FTS3Agent working only with the FTS3 service to replace the existing one
NEW: FTS3Utilities - use correct FTS Server Selection Policy
*Resources
NEW: New SingularityComputingElement to submit jobs to a Singularity container
*Docs
FIX: Drop some old stuff, add link for FTS3 page
[v6r19p3]
CHANGE: .pylintrc - disable redefined-variable-type
CHANGE: .pylintrc - max-nested-blocks=10 due to the many tests of result['OK']
CHANGE: use autopep8 for auto-formatting with following exceptions:
tabs = 2 spaces and not 4
line length check disabled (i.e. 120 characters instead of 80)
Option for autopep8 are: --ignore E111,E101,E501
*Configuration
FIX: retrigger the initialization of the logger and the ObjectLoader after
all the CS has been loaded
*WMS
FIX: pilot commands will add /DIRAC/Extensions=extensions if requested
FIX: SiteDirector, pilotCommands - fix support for multiple values in the
RequiredTag CE parameter
FIX: MultiProcessorSiteDirector - fix dictionary changed size exception
*Workflow
FIX: application log name can also come from step_commons.get['logFile']
*Resources
CHANGE: Condor, SLURM, SSHComputingElement - added parameters to force allocation
of multi-core job slots
[v6r19p2]
*DMS
FIX: dirac-admin-allow-se: fix crash because of usage of old RSS function
*RSS
FIX: ResourceStatusDB - microseconds should always be 0
FIX: Multiple fixes for the RSS tests
[v6r19p1]
*Core
FIX: ElasticSearchDB - certifi package was miscalled
FIX: ElasticSearchDB - added debug messages for DB connection
*Framework
FIX: ComponentInstaller - handling correctly extensions of DBs found in sql files
*WMS
FIX: SudoComputingElement - prevent message overwriting application errors
FIX: JobDB.getInputData now returns list of cleaned LFNs strings, possible "LFN:"
prefix is removed
*Interfaces
FIX: Dirac.py - bring back treatment of files in working local submission directory
[v6r19]
FIX: In multiple places - use systemCall() rather than shellCall() to avoid
potential shell injection problems
FIX: All Databases are granting also REFERENCES grant to Dirac user to comply with
more strict policies of MySQL version >= 5.7
*Accounting
NEW: new functionality to plot the data gathered by perfSONARs. It allows to
present jitter, one-way delay, packet-loss rate and some derived functions.
FIX: compatibility of AccountingDB with MySQL 5.7
*ConfigurationSystem
NEW: Allow to define FailoverURLs and to reference MainServers in the URLs
*FrameworkSystem
NEW: gLogger is replaced by the new logging system based on the python logging module
NEW: Added ElasticSearch backend for the logging
NEW: Central Backends configuration to customize their use by multiple components
NEW: BundleDelivery - serves also CA's and CRL's all-in-one files
NEW: added shell scripts for generating CAs and CRLs with the possibility to specify the Input and/or output directories
CHANGE: can now send mails to multiple recipients using the NotificationClient
CHANGE: Make the new logging system thread-safe
FIX: Adapting query to MySLQ 5.7 "GROUP BY" clause
FIX: TopErrorMessagesReporter - more precise selection to please stricter versions of MySQL
CHANGE: ProxyGeneration - make RFC proxies by default, added -L/--legacy flag to dirac-proxy-init
to force generation of no-RFC proxies
*Core
FIX: dirac-install - allow to use local md5 files
CHANGE: X509Chain - fixes to allow robot proxies with embedded DIRAC group extension
( allow DIRAC group extension not in the first certificate chain step )
CHANGE: BaseClient - recheck the useServerCertificate while establishing connection
and take it into account even if it has changed after the client object creation
FIX: PlainTransport - fixed socket creation in initAsClient()
NEW: Technology preview of new logging system, based on standard python logging module
CHANGE: Added graphviz extension to sphinx builds
FIX: Added documentation of low level RPC/DISET classes
FIX: Gateway service - multiple fixes to resurrect the service and to correctly instantiate it
NEW: dirac-install will change the shebang of the python scripts to use the environment
python instead of the system one
NEW: Security.Utilities - methods to generate all-in-one CA certificates and CRLs files
NEW: ElasticSearchDB - gets CA's all-in-one file from the BundleDelivery service if needed
NEW: genAllCAs.sh, genRevokedCerts.sh - DIRAC-free commands to generate all-in-one CA
certificates and CRLs files
CHANGE: dirac-create-distribution-tarball - removing docs and tests directories when
creating release tarballs
*DMS
CHANGE: FTSJob - use Request wrapper for the fts3 REST interface instead of pycurl based
client
CHANGE: FTSHistoryView - drop FTSServer field from the view description
CHANGE: FTSFile DB table: increased length of fields LFN(955), SourceSURL(1024), TargetSURL(1024)
CHANGE: Uniform length of LFN to 255 across DIRAC dbs
FIX: FTSJob - fix the serialization of 0 values
FIX: FTSFile, FTSJob - fix SQL statement generation for stricter versions of MySQL
*Resources
NEW: New method in the StorageElement to generate pair of URLs for third party copy.
Implement the logic to generate pair of URLs to do third party copy.
This will be used mostly by FTS, but is not enabled as of now
FIX: StorageElement - fix different weird behaviors in Storage Element, in particular,
the inheritance of the protocol sections
FIX: GFAL2 storage element: update for compatibility with GFAL2 2.13.3 APIs
NEW: Introduced Resources/StorageElementBases configuration section for definitions
of abstract SEs to be used in real SEs definition by inheritance
*RMS
NEW: dirac-rms-request - command including functionality of several other commands:
dirac-rms-cancel|reset|show-request which are dropped. The required functionality
is selected by the appropriate switches
*RSS
NEW: Put Sites, ComputingElements, FTS and Catalogs under the status control of the
RSS system
NEW: Rewrote RsourceStatus/ResourceManagementDB tables with sqlAlchemy (RM DB with declarative base style)
NEW: SiteStatus client to interrogate site status with respect to RSS
CHANGE: introduced backward compatibility of RSS services with DIRAC v6r17 clients
CHANGE: moved some integration tests from pytest to unittest
CHANGE: Moved ResourceStatusDB to sqlAlchemy declarative_base
FIX: Automated setting of lastCheckTime and Dateffective in ResourceStatusDB and ResourceManagementDB
FIX: fixes for tables inheritance and extensions
FIX: fixes for Web return structure ("meta" column)
FIX: ResourceStatus, RSSCacheNoThread - fixed RSS cache generation
FIX: ResourceStatus - fixes for getting status from the CS information
FIX: ResourceManagement/StatusDB - fixed bugs in meta parameter check
FIX: fixed incompatibility between Active/InActive RSS clients return format
FIX: SiteStatus - bug fixed in getSites() method - siteState argument not propagated to
the service call
FIX: ResourceStatus - return the same structure for status lookup in both RSS and CS cases
FIX: Bug fixes in scripts getting data out of DB
*Monitoring
CHANGE: DBUtils - change the bucket sizes for the monitoring plots as function of the time span
*WMS
NEW: SiteDirector - checks the status of CEs and Sites with respect to RSS
NEW: pilotCommands - new ReplaceDIRACCode command mostly for testing purposes
NEW: JobAgent, JobWrapper - several fixes to allow the work with PoolComputingElement
to support multiprocessor jobs
NEW: JobScheduling - interpret WholeNode and NumberOfProcessors job JDL parameters and
convert then to corresponding tags
NEW: SiteDirector - CEs can define QueryCEFlag in the Configuration Service which can be
used to disallow querying the CE status and use information from PiltAgentsDB instead
NEW: The application error codes, when returned, are passed to the JobWrapper, and maybe interpreted.
NEW: The JobWrapperTemplate can reschedule a job if the payload exits with status DErrno.EWMSRESC & 255 (222)
FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system
FIX: JobDB - fixes necessary to suite MySQL 5.7
FIX: dirac-pilot, pilotTools - PYTHONPATH is cleared on pilot start, pilot option keepPP
can override this
FIX: WMSAdministratorHandler - make methods static appropriately
FIX: Bug fix for correctly excluding WebApp extensions
CHANGE: JobScheduling - more precise site name while the job is Waiting, using the set of
sites at which the input files are online rather than checking Tier1s in eligible sites
FIX: SiteDirector - aggregate tags for the general job availability test
FIX: JobScheduling - bug fix in __sendToTQ()
FIX: pilotTools,pilotCommands - pick up all the necessary settings from the site/queue configuration
related to Tags and multi-processor
NEW: SiteDirector - added option to force lcgBundle version in the pilot
FIX: SiteDirector - if MaxWaitingJobs or MaxTotalJobs not defined for a queue, assume a default value of 10
FIX: MatcherHandler - preprocess resource description in getMatchingTaskQueues()
FIX: JobDB - set CPUTime to a default value if not defined when rescheduling jobs
*TS
FIX: TransformationClient - fix issue #3446 for wrong file error counting in TS
FIX: TransformationDB - set ExternalID before ExternalStatus in tasks
BUGFIX: TransformationClient - fix a bug in the TS files state machine (comparing old status.lower()
with new status)
*Interfaces
CHANGE: Dirac API - expose the protocol parameter of getAccessURL()
CHANGE: Dirac API - added runLocal as an API method
*Docs
NEW: Documentation for developing with a container (includes Dockerfile)
NEW: Add script to collate release notes from Pull Request comments
NEW: Chapter on scaling and limitations
CHANGE: Added documentation about runsv installation outside of DIRAC
*tests
NEW: Added client (scripts) system test
CHANGE: Add to the TS system test, the test for transformations with meta-filters
FIX: Minor fixes in the TS system test
FIX: correctly update the DFC DB configuration in jenkins' tests
[v6r17p35]
*Core
FIX: GOCDBClient - add EXTENSIONS & SCOPE tag support to GOCDB service queries.
[v6r17p34]
*SMS
FIX: StorageManagerClient - fix logic for JobScheduling executor when CheckOnlyTapeSEs is
its default true and the lfn is only on a tapeSE
[v6r17p33]
*WMS
FIX: StalledJobAgent - if no PilotReference found in jobs parameters, do as if there would be
no pilot information, i.e. set Stalled job Failed immediately
CHANGE: DownloadInputData - job parameters report not only successful downloads but also failed ones
FIX: JobDB - back port - set CPUTime to 0 if not defined at all for the given job
FIX: JobDB - back port - use default CPUTime in the job description when rescheduling jobs
*Resources
FIX: ARCComputingElement - fix job submission issue due to timeout for newer lcg-bundles
[v6r17p32]
Resources:
CHANGE: /Computing/BatchSystems/Condor.py: do not copy SiteDirector's shell environment variables into the job environment
*WMS
CHANGE: Add option to clear PYTHONPATH on pilot start
[v6r17p31]
*RMS
FIX: ReqClient - avoid INFO message in client
*WMS
CHANGE: JobWrapper - allow SE-USER to be defined as another SE group (e.g. Tier1-USER)
*DMS
CHANGE: DMSHelpers - make resolveSEGroup recursive in order to be able to define SE groups in terms of SE groups
[v6r17p30]
*DMS
CHANGE: StorageElement - added status(), storageElementName(), checksumType() methods returning
values directly without the S_OK structure. Remove the checks of OK everywhere
NEW: dirac-dms-add-file, DataManager - added option (-f) to force an overwrite of an existing file
*TS:
FIX: TransformationDB.py - set the ExternalID before the ExternalStatus in order to avoid inconsistent
tasks if setting the ExternalID fails
*StorageManagementSystem
FIX: StorageManagementClient.py - return the full list of onlineSites while it was previously happy
with only one
*Resources
FIX: HTCondorCEComputingElement.py - transfer output files(only log and err) for remote scheduler
[v6r17p29]
*WMS
CHANGE: split time left margins in cpuMargin and wallClockMargin. Also simplified check.
[v6r17p28]
*WMS
BUGFIX: JobScheduling - fix a bug introduced in 6r17p27 changes
*Monitoring
BUGFIX: MonitoringReporter - do not try to close the MQ connection if MD is not used
[v6r17p27]
*Configuration
FIX: ConfigurationClient - allow default value to be a tuple, a dict or a set
*Monitoring
CHANGE: DBUtils - change bucket sizes and simplify settings
*DMS
FIX: DMSRequestOperationsBase, RemoveFile - allow request to not fail if an SE is temporarily banned
FIX: dirac-admin-allow-se - first call of gLogger after its import
*RMS
CHANGE: remove scripts dirac-rms-show-request, dirac-rms-cancel-request and dirac-rms-reset-request
and replace with a single script dirac-rms-request with option (default is "show")
CHANGE: allow script to finalize a request if needed and set the job status appropriately
*Resources
FIX: LocalComputingElement - pilot jobIDs start with ssh to be compatible with pilotCommands.
Still original jobIDs are passed to getJobStatus. To be reviewed
*WMS
CHANGE: JobScheduling - assign a job to Group.<site>.<country>, if input files are at <site>.<country>.
If several input replicas, assign Waiting to "MultipleInput"
[v6r17p26]
*Core
FIX: dirac-install.py to fail when installation of lcgBundle has failed
FIX: ClassAdLight - getAttributeInt() and getAttributeFloat() return None
if the corresponding JDL attribute is not defined
*MonitoringSystem
CHANGE: The Consumer and Producer use separate connections to the MQ;
If the db is not accessible, the messaged will not be consumed.
*WMS
FIX: JobDB - fix the case where parametric job placeholder %j is used in the JobName attribute
FIX: JobDB - take into account that ClassAdLight methods return None if numerical attribute is not defined
FIX: ParametricJob utility - fixed bug in evaluation of the ParameterStart|Step|Factor.X job numerical attribute
[v6r17p25]
*Monitoring
NEW: Implemented the support of monthly indexes and the unit tests are fixed
*RMS
FIX: RequestExecutingAgent - fix infinite loop for duplicate requests
*WMS
NEW: ARCComputingElement - add support for multiprocessor jobs
[v6r17p24]
*WMS
FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system
[v6r17p23]
*WMS
FIX: get job output for remote scheduler in the case of HTCondorCE
[v6r17p22]
*Framework
FIX: NotificationClient - added avoidSpam flag to sendMail() method which is propagated to
the corresponding service call
*Integration
FIX: several fixes in integration testing scripts
[v6r17p21]
*Core
NEW: Mail.py - added mechanism to compare mail objects
FIX: Grid.py - take into account the case sometimes happening to ARC CEs
where ARC-CE BDII definitions have SubClusters where the name isn't set to
the hostname of the machine
*Framework
FIX: Notification service - avoid duplicate emails mechanism
[v6r17p20]
*Core
NEW: API.py - added __getstate__, __setstate__ to allow pickling objects inheriting
API class by special treatment of internal Logger objects, fixes #3334
*Framework
FIX: SystemAdministrator - sort software version directories by explicit versions in the
old software cleaning logic
FIX: MonitoringUtilities - sets a suitable "unknown" username when installing DIRAC from scratch,
and the CS isn't initialized fully when running dirac-setup-site
CHANGE: Logger - added getter methods to access internal protected variables, use these methods
in various places instead of access Logger protected variables
*WMS
CHANGE: JobDB - removed unused CPUTime field in the Jobs table
CHANGE: JobScheduling - make check for requested Platform among otherwise eligible sites
for a given job, fail jobs if no site with requested Platform are available
*RSS
FIX: Commands - improved logging messages
*SMS
FIX: StorageManagerClient - instantiate StorageElement object with an explicit vo argument,
fixes #3335
*Interfaces
NEW: dirac-framework-self-ping command for a server to self ping using it's own certificate
[v6r17p19]
*Core
FIX: Adler - fix checksum with less than 8 characters to be 8 chars long
*Configuration
FIX: VOMS2CSAgent - fix to accomodate some weird new user DNs (containing only CN field)
*DMS
FIX: FileCatalog - fix for the doc strings usage in file catalog CLI, fixes #3306
FIX: FileCatalog - modified recursive file parameter setting to enable usage of the index
*SMS
CHANGE: StorageManagerClient - try to get sites with data online if possible in getFilesToStage
*RMS
FIX: RequestExecutingAgent - tuning of the request caching while execution
*WMS
FIX: DownloadInputData - do not mistakenly use other metadata from the replica info than SEs
FIX: JobScheduling - put sites holding data before others in the list of available sites
FIX: JobScheduling - try and select replicas for staging at the same site as online files
FIX: SiteDirector - keep the old pilot status if the new one can not be obtained in updatePilotStatus()
*Resources
FIX: CREAMComputingElement - return error when pilot output is missing in getJobOutput()
*Monitoring
FIX: DBUtils - change the buckets in order to support queries which require more than one year
data. The maximum buckets size is 7 weeks
[v6r17p18]
*Framework
NEW: SystemAdministrator - added possibility to remove old software installations keeping
only a predefined number of the most recent ones.
*DMS
FIX: RemoveReplica - removing replica of a non-existing file is considered successful
*SMS
CHANGE: StorageManagerClient - restrict usage of executeWithUserProxy decorator
to calling the SE.getFileMetadata only; added flag to check only replicas
at tape SEs
*WMS
FIX: JobScheduling - added CS option to flag checking only replicas at tape SEs;
fail jobs with input data not available in the File Catalog
[v6r17p17]
*DMS
NEW: FTSAgent has a new CS parameter ProcessJobRequests to be able to process job
requests only. This allows to run 2 FTS agents in parallel
*Resources
FIX: GFAL2_StorageBase - only set the space token if there is one to avoid problems
with some SEs
[v6r17p16]
*Configuration
FIX: VOMS2CSAgent - create user home directory in the catalog without
recursion in the chown command
*RMS
FIX: RequestExecutingAgent - catch error of the cacheRequest() call
FIX: ReqClient - enhanced log error message
*SMS
FIX: StorageManagerClient - treat the case of absent and offline files on an SE
while staging
*TS
FIX: TaskManagerBase - process tasks in chunks of 100 in order to
update faster the TS (tasks and files)
*WMS
FIX: JobScheduling - do not assume that all non-online files required staging
[v6r17p15]
*WMS
CHANGE: StalledJobAgent - ignore or prolong the Stalled state period for jobs
at particular sites which can be suspended, e.g. Boinc sites
[v6r17p14]
*Core
FIX: PrettyPrint.printTable utility enhanced to allow multi-row fields and
justification specification for each field value
*Accounting
NEW: DataStore - allow to run several instances of the service with only one which
is enabled to do the bucketing
*RMS
NEW: new dirac-rms-list-req-cache command to list the requests in the ReqProxies services
*Interfaces
CHANGE: Dirac API - make several private methods visible to derived class
[v6r17p13]
*Core
NEW: Proxy - added executeWithoutServerCertificate() decorator function
*Resources
FIX: CREAMComputingElement - split CREAM proxy renewal operation into smaller chunks for
improved reliability
[v6r17p12]
*Framework
FIX: SecurityFileLog - when the security logs are rotated, the buffer size is reduced
to 1 MB to avoid gzip failures ( was 2 GBs )
*WMS
FIX: pilotCommands - fix for interpreting DNs when saving the installation environment
FIX: SandboxStoreClient - do not check/make destination directory if requested sandbox
is returned InMemory
*TS
FIX: TransformationAgent CS option MaxFiles split in MaxFilesToProcess and MaxFilesPerTask,
MaxFiles option is interpreted as MaxFilesPerTask for backward compatibility
*Resources
NEW: Added plug-ins for GSIFTP and HTTPS Storage protocols
[v6r17p11]
*Core
FIX: ElasticSearchDB - set a very high number (10K) for the size of the ElasticSearch result
*Monitoring
FIX: MonitoringDB - et a very high number (10K) for the size of the ElasticSearch result
*WMS
FIX: pilotCommands - get the pilot environment from the contents of the bashrc script
*DMS
FIX: RemoveReplica - fix for the problem that if an error was set it was never reset
FIX: SE metadata usage in several components: ConsistencyInspector, DataIntwgrityClient,
FTSRequest, dirac-dms-replica-metadata, StageMonitorAgent, StageRequestAgent,
StorageManagerClient, DownloadInputData, InputDataByProtocol
[v6r17p10]
*Core
NEW: Logger - printing methods return True/False if the message was printed or not
FIX: ElastocSearchDB - error messages demoted to warnings
*Monitoring
FIX: MonitoringReporter - create producers if the CS definitions are properly in place
*TS
CHANGE: TaskManagerPlugin - allow to redefine the AutoAddedSites for each job type
[v6r17p9]
*WMS
BUGFIX: JobScheduling - bug fixed introduced in the previous patch
NEW: pilotTools - introduced -o swicth for a generic CS option
*SMS
FIX: StorageManagerClient - fixes in the unit test
*DMS
FIX: FileManagerPs - in _getFileLFNs() - break a long list of LFNs into smaller chunks
[v6r17p8]
*Core
NEW: DErrno.ENOGROUP error to denote proxies without DIRAC group extension embedded
CHANGE: X509Chain - use DErrno.ENOGROUP error
FIX: dirac-install, dirac-deploy-scripts - fixes to allow DIRAC client installation on
recent MacOS versions with System Integrity Protection feature
CHANGE: Proxy - added executionLock optional argument to executeWithUserProxy() decorator
to lock while executing the function with user proxy
FIX: Proxy - fix indentation in getProxy() preventing looping on the DNs
*Framework
FIX: ProxyDB - fix of error message check in completeDelegation()
*WMS
FIX: TaskQueueDB - when an empty TaskQueue is marked for deletion, it can still get matches
which result in no selected jobs that produced unnecessary error messages
FIX: JobScheduling executor - calls getFilesToStage() with a flag to lock while file lookup
with user proxy; same for InputData executor for calling _resolveInputData()
*TS
FIX: FileReport - fix in setFileStatus() for setting status for multiple LFNs at once
*SMS
FIX: StorageManagerClient - in getFilesToStage() avoid using proxy if no files to check
on a storage element
*Resources
FIX: GFAL2_XROOTStorage - fix to allow interactive use of xroot plugin
FIX: GFAL2_StorageBase - enable IPV6 for gsiftp
[v6r17p7]
*DMS
FIX: dirac-dms-user-lfns - do not print out empty directories
*WMS
FIX: InputData Executor, JobWrapper - use DataManager.getReplicasForJobs() for
getting input data replicas
*TS
FIX: TransformationAgent - use DataManager.getReplicasForJobs() for transformations
creating jobs
[v6r17p6]
*DMS
NEW: DataManager - add key argument forJobs (default False) in getReplicas() in order
to get only replicas that can be used for jobs (as defined in the CS); added
getReplicasForJobs(), also used in the Dirac API
*SMS
FIX: Stager agents - monitor files even when there is no requestID, e.g. dCache returns None
when staging a file that is already staged
*Resources
FIX: StorageFactory - bug fixes when interpreting SEs inheriting other SE parameters
NEW: Test_StorageFactory unit test and corresponding docs
FIX: Torque - some sites put advertising in the command answer that can not be parsed:
redirect stderr to /dev/null
[v6r17p5]
*Resources
FIX: LcgFileCatalogClient - do not evaluate GUID if it is not a string
[v6r17p4]
*Configuration
FIX: Utilities - fixed interpretation of weird values of GlueCEPolicyMaxWallClockTime
BDII parameter; newMaxCPUTime should is made integer
*Framework
FIX: Logger - make subloggers processing messages with the same level
as the parent logger
*Docs
NEW: Updated documentation in several sections
*DMS
FIX: RemoveReplica operation - don't set file Done in RemoveReplicas if there is an error
[v6r17p3]
*RSS
FIX: Synchronizer - the sync method removes the resources that are no longer
in the CS from the DowntimeCache table
*DMS
CHANGE: dirac-dms-find-lfns - added SE switch to look for files only having
replicas on a given SE (list)
*TS
FIX: TaskManager - optimization of the site checking while preparing job; optimized
creation of the job template
*Resources
CHANGE: GFAL2_SRM2Storage, SRM2Storage - added gsiftp to the list of OUTPUT protocols
[v6r17p2]
*Monitoring
FIX: ElasticSearchDB - fixes required to use host certificate for connection;
fixes required to pass to version 5.0.1 of the elasticsearch.py binding
[v6r17p1]
*RSS
FIX: GOCDBSync - make commmand more verbose and added some minor fixes
[v6r17]
*Core
FIX: Adler - check explicitly if the checksum value is "False"
FIX: install_site.sh - added command line option to choose DIRAC version to install
NEW: ComponentInstaller - added configuration parameters to setup NoSQL database
*Framework
CHANGE: Logger - test level before processing string (i.e. mostly converting objects to strings)
CHANGE: dirac-proxy-init - check and attempt to update local CRLs at the same time as
generating user proxy
CHANGE: ProxyManager service - always store the uploaded proxy even if the already stored
one is of the same validity length to allow replacement in case of proxy type
changes, e.g. RFC type proxies
*DMS
NEW: Next in implementation multi-protocol support for storage elements. When performing
an action on the StorageElement, instead of looping over all the protocol plugins,
we loop over a filtered list. This list is built taking into account which action
is taken (read vs write), and is also sorted according to lists defined in the CS.
The negotiation for third party transfer is also improved: it takes into account all
possible protocols the source SE is able to produce, and all protocols the target is
able to receive as input.
NEW: StorageElement - added methods for monitoring used disk space
FIX: ReplicateAndRegister - fix the case when checksum is False in the FC
NEW: DMSHelpers - get list of sites from CS via methods; allow to add automatically sites
with storage
*RSS
NEW: FreeDiskSpace - added new command which is used to get the total and the remaining
disk space of all dirac storage elements that are found in the CS and inserts the
results in the SpaceTokenOccupancyCache table of ResourceManagementDB database.
NEW: GOCDBSync command to ensure that all the downtime dates in the DowntimeCache
table are up to date
Resources*
NEW: Updated Message Queue interface: MQ service connection management, support for
SSL connections, better code arrangement
*Workflow
FIX: Modulebase, Script - avoid too many unnecessarily different application states
*WMS
FIX: JobStateUpdate service - in setJobStatusBulk() avoid adding false information when adding
an application status
*TS
FIX: TaskManager, TaskManagerAgentBase - standardize the logging information; removed unnecessary
code; use iterators wherever possible
NEW: Introduced metadata-based filters when registering new data in the TS as catalog
[v6r16p6]
*WMS
NEW: Added MultiProcessorSiteDirector section to the ConfigTemplate.cfg
*DMS
FIX: FileCatalogClient - added missing read methods to the interface description
getDirectoryUserMetadata(), getFileUserMetadata()
[v6r16p5]
FIX: included patches from v6r15p27
[v6r16p4]
FIX: applied fixes from v6r15p26
[v6r16p3]
FIX: incorporated fixes from v6r15p25
[v6r16p2]
*Configuration
CHANGE: VOMS2CSAgent - remove user DNs which are no more in VOMS. Fixes #3130
*Monitoring
CHANGE: WMSHistory - added user, jobgroup and usergroup selection keys
*DMS
FIX: DataManager - retry checksum calculation on putAndRegister, pass checksum to the DataManager
object in the FailoverTransfer object.
FIX: DatasetManager, FileCatalogClientCLI - bug fixes in the dataset management and commands
*WMS
CHANGE: JobManager - added 'Killed' to list of jobs status that can be deleted
[v6r16p1]
*Monitoring
CHANGE: MonitorinDB - allow to use more than one filter condition
*WMS
CHANGE: StalledJobAgent - send a kill signal to the job before setting it Failed. This should
prevent jobs to continue running after they have been found Stalled and then Failed.
[v6r16]
*Core
CHANGE: dirac-install, dirac-configure - use Extensions options consistently, drop
ExtraModule option
CHANGE: dirac-install - use insecure ssl context for downloading files with urllib2.urlopen
CHANGE: GOCDBClient - replaced urllib2 with requests module
FIX: dirac-setup-site - added switch to exitOnError, do not exit on error by default
CHANGE: Added environment variables to rc files to enable certificates verification (necessary for python 2.7.9+)
FIX: ComponentInstaller - always update CS when a database is installed, even if it is
already existing in the db server
FIX: SSLSocketFactory - in __checkKWArgs() use correct host address composed of 2 parts
*Framework
FIX: SystemAdministrator service - do not install WebAppDIRAC by default, only for the host
really running the web portal
*Accounting
FIX: JobPolicy - remove User field from the policy conditions to fix a problem that
non-authenticated user gets more privileges on the Accounting info.
*Monitoring
NEW: New Monitoring system is introduced to collect, analyze and display various
monitoring information on DIRAC components status and behavior using ElasticSearch
database. The initial implementation is to collect WMSHistory counters.
*DMS
NEW: MoveReplica operation for the RMS system and a corresponding dirac-dms-move-replica-request
comand line tool
*Resources
NEW: MessageQueue resources to manage MQ connections complemented with
MQListener and MQPublisher helper classes
NEW: SudoComputingElement - computing element to execute payload with a sudo to a dedicated
UNIX account
[v6r15p27]
*Configuration
FIX: CSAPI - changed so that empty but existing options in the CS can be still
modified
[v6r15p26]
*WMS
FIX: SandboxStoreClient - ensure that the latest sandbox is returned in the Web
portal in the case the job was reset.
[v6r15p25]
*Resources
FIX: HTCondorCEComputingElement - cast useLocalSchedd to bool value even if it
is defined as srting
[v6r15p24]
*Resources
CHANGE: HTCondorCE - added option to use remote scheduler daemon
[v6r15p23]
*DMS
FIX: dirac-dms-find-lfns - fixed bug causing generl script failure
[v6r15p22]
*Interfaces
CHANGE: Dirac API - add possibility to define the VO in the API
CHANGE: Dirac API - add checkSEAccess() method for checking SE status
[v6r15p21]
*WMS
FIX: removed default LCG version from the pilot (dirac-install will use the one of the requested release)
*RMS
FIX: reject bad checksum
[v6r15p20]
*Framework
FIX: SystemAdministratorHandler - in updateSoftware() put explicitly the project
name into the command
FIX: ComponentInstaller - added baseDir option to the mysql_install_db call
while a fresh new database server installation
[v6r15p19]
*Core
FIX: dirac-install - lcg-binding version specified in the command switch
overrides the configuration option value
*DMS
FIX: RemoveFile operation - Remove all files that are not at banned SEs
*TMS
FIX: FileReport - after successful update of input files status, clear the
cache dictionary to avoid double update
[v6r15p18]
*Configuration
FIX: Utilities - take into account WallClock time limit while the MaxCPUTime
evaluation in the Bdii@CSAgent
*DMS
FIX: FTSJob - specify checksum type at FTS request submission
*StorageManagement
FIX: StorageManagerClient - in getFilesToStage() avoid exception in case
of no active replicas
*Resources
FIX: StorageBase - in getParameters() added baseURL in the list of parameters returned
*WMS
FIX: CPUNormalization - minor code rearrangement
[v6r15p17]
*Core
CHANGE: GOCDBClient - catch all downtimes, independently of their scope
FIX: LSFTimeLeft - accept 2 "word" output from bqueues command
CHANGE: dirac-install - create bashrc/cshrc with the possibility to define
installation path in the $DIRAC env variable, this is needed for
the cvmfs DIRAC client installation
[v6r15p16]
*Core
CHANGE: AgentModule - added a SIGALARM handler to set a hard timeout for each Agent
cycle to avoid agents stuck forever due to some faults in the execution code
*DMS
FIX: DataManager - cache SE status information in filterTapeReplicas() to speed up execution
*WMS
BUGFIX: InputDataByProtocol - the failed resolution for local SEs was not considered correctly:
if there were other SEs that were ignored (e.g. because on tape)
*TS
FIX: TransformationAgent - in getDataReplicasDM() no need to get replica PFNs
[v6r15p15]
*Configuration
CHANGE: VOMS2CSAgent - added new features: deleting users no more registered in VOMS;
automatic creation of home directories in the File Catalog for new users
*WMS
CHANGE: JobScheduling - correct handling of user specified sites in the executor,
including non-existent (misspelled) site names
FIX: CPUNormalization - accept if the JOBFEATURES information is zero or absent
[v6r15p14]
*Core
FIX: BaseClient - proper error propagation to avoid excessive output in the logger
*Configuration
CHANGE: Resources helper - in getStorageElementOptions() dereference SEs containing
BaseSE and Alias references
*Accounting
FIX: AccountingDB - changes to use DB index to speed-up removal query
*DMS
CHANGE: DMSHelpers - define SE groups SEsUsedForFailover, SEsNotToBeUsedForJobs,
SEsUsedForArchive in the Operations/DataManagement and use them in the
corresponding helper functions
FIX: FTSJob - temporary fix for the FTS rest interface Request object until it is
fixed in the FTS REST server
*Resources
FIX: HTCondorCEComputingElement - check that some path was found in findFile(), return with error otherwise
CHANGE: ARCComputingElement - consider jobs in Hold state as Failed as they never come back
CHANGE: ARCComputingElement - do not use JobSupervisor tool for bulk job cancellation as
it does not seem to work, cancel jobs one by one
FIX: ARCComputingElement - ensure that pilot jobs that are queued also get their proxies renewed on ARC-CE
*WMS
FIX: SiteDirector - ensure that a proxy of at least 3 hours is available to the updatePilotStatus
function so that if it renews any proxies, it's not renewing them with a very short proxy
[v6r15p13]
*Resources
FIX: HTCondorCEComputingElement - fixed location of log/output files
*TS
FIX: ValidateOutputDataAgent - works now with the DataManager shifter proxy
[v6r15p12]
*Core
FIX: Graphs - make sure matplotlib package is always using Agg backend
FIX: cshrc - added protection for cases with undefined environment variables
NEW: AuthManager - added possibility to define authorization rules by VO
and by user group
*Configuration
NEW: Resources, ComputingElement(Factory) - added possibility to define site-wide
CE parameters; added possibility to define common parameters for a given
CE type.
*Framework
FIX: SystemAdministrator service - avoid using its own client to connect
to itself for storing host information
FIX: SystemAdministratorClientCLI, dirac-populate-component-db - fix insertion
of wrongly configured component to the ComponentMonitorDB
*DMS
FIX: FileCatalog service - fix the argument type for getAncestor(), getDescendents()
*WMS
NEW: JobCleaningAgent - add an option (disabled by default) to remove Jobs from the
dirac server irrespective of their state