Blog Viewer

Summary:

The end-to-end lifecycle below is a detailed and technical explanation of inbound file processing. Unless you are going to override any of the default file processing methods or configure any of the default S3 file processing flags, you only need to know the following:

• The application follows a process defined by the framework to validate and upload and then process any inbound file.
• Although this process is already in place and functional in the application as part of the base package, you can override the methods that do the processing at any stage to customize it for your specific inbound file.
• The end result of processing an inbound file is that it has been routed to the appropriate system folder based on validation results, and the relevant screens have been updated (errors permitting).

If you want to customize processing an inbound file, please read about the lifecycle below.

End-to-End Inbound File Processing Lifecycle:

1. The application validates and uploads the file.
   1. It checks if the HeaderGroupArray is null and HeaderGroupPosition is greater than zero. If true, it initializes the HeaderGroup position and length array.
      1. The application calls the InitializeFile method. This is defined in the framework, and you can override it if you want it to do any specific task. Method parameters are file header, file layout, and caller name (acdoFileHdr, autlFileLayout, astrCallerName = null). Caller Name helps to identify the file process like validate and upload or process file.
   2. It checks if the file validate and upload is parallel. If true, then it calls the process ValidateAndUploadFileParallel. If false, it calls the process ValidateAndUploadFileSequential.
      1. In ValidateAndUploadFileParallel, it checks for GetGroupValueFromHeader. If true, it populates the group values.
         1. It reads each record. For each record, it: 
            1. Sets the transaction code. This can be overridden with the method SetTransactionCode.
            2. Call the method IgnoreRecord. This is defined in the framework and you can override it. If true, it continues with the next record. If false, it calls the SetHeaderGroupValue method. This is defined in the framework and you can override it. It assigns the HeaderGroupValue to the record number and returns the HeaderGroupValue.
         2. The application calls the IsFileValid method. This can be overridden if you want it to do any specific task.
            1. If the file extension returns an error message, the application updates the File Header with that message and sets status to Review. Once all the records are validated, the application calls the AfterRecordsValidation method. This is defined in the framework and you can override it on the business object if you want to add or remove errors.
            2. If the file extension does not return an error message, the application checks every record for unsafe characters. If it finds any, it calls the AllowUnsafeRecord method and either takes the modified record or rejects the file. The framework takes the unsafe character from the system set table against configuration value. You can override it in the business object if you modify the record and provide it as a modified record for further processing.
         3. The application prepares the UTLFRParams.
            1. The application calls the SetTransactionCode method. This can be overridden. It sets the transaction code. If the transaction_code_pos value is 0, then the framework assigns the transaction code value as NONE. Transaction code is set based on the transaction code position value. If there is no transaction code, the application raises a 5004 message and calls the CheckErrorOnUpload method, which calls the IgnoreErrorOnUpload method This can be overridden at the project level and you can decide whether the record needs to be ignored or not based on the return value. If the return is false, the application logs the error in the file detail error table and updates the status to Review. If the return is true, it continues with the next record.
            2. The application calls the IgnoreRecord method. This is defined in the framework and you can override it and get the file layout. This method helps to ignore specific records from the file based on the business requirements. To ignore a record, override the function needed to return a value of true.
            3. The application gets the file layout. If it can't get the file layout, you can override with the IgnoreErrorOnUpload method. This raises the 5006 message.
         4. The application gets the field list for fixed length. If fixed length is less than the field length and record length, it raises the 5001 message.
         5. The application calls the SetHeaderGroupValue method, which can be overridden in the business object. This sets the header_group_value for the file detail if the header group is based on multiple fields.
         6. Based on S3 settings, (sfwUploadToFileDetail/sfwReceiveToFileDetail), the application creates file detail entries.
         7. The application calls the BasicValidation method. It checks if there is any error on upload at field level and performs validations at business level logic, which can be done by overriding the CurrentRecordValidation method.
            1. If the application finds no errors, it creates the object based on the record:
               1. Summary: NewSummary method. It can be overridden.
               2. Header: SetHeaderGroupValue and NewHeaderGroup methods. Both can be overridden.
               3. Detail: NewDetail method.
            2. The application binds to the object.
               1. For the Record Type, it checks for a header or summary. If there is none, it calls the NewDetail method, which can be overridden. This creates the new object based on the object ID mapping in the XML file.
               2. For each field mapping, it calls the BeforeFieldAssigned method, which can be overridden. This method tries to set the value based on mapping. If there is an error, the application calls the ContinueOnValueError method on the file business object. You can override this, but it has to return one of the following:
                  1. SkipRecord (utlOnRecordError.ContinueSkipRecord): The record is skipped and the application starts processing the next record after adding the error to sgs_file_dtl. If the skip record condition is true, the status is assigned as Ignore. If the condition is false, the status is assigned as Review.
                  2. WithRecord (utlOnRecordError.ContinueWithRecord): The error is added to sgs_file_dtl_error. It assigns the property based on the modified value.
                  3. RejectFile (utlOnRecordError.RejectFile): It adds the error message and the file is rejected.
               3. As part of the process on upload, it overrides the IgnoreErrorOnUpload method.
            3. The application calls the ValidateHardError method and for each error, calls the CheckErrorOnUpload method. The application sets the file status accordingly.
               1. Once all the records have been validated, the application calls the AfterRecordsValidation method. This is defined in the framework, and you can override it on the business object if you want to add or remove errors. 
               2. The framework checks whether validate hard error is true based on the sfwValidateHardErrorsOnUpload value from S3. This field gets assigned against each transaction type. If yes, then the framework executes the BeforeValidate and ValidateHardErrors methods. These perform validation of constraints and validation rules. You can override these methods at the business level.
            4. The application checks if it is a valid record. If valid record condition is false, then it will increment the error count.
            5. The application commits the transaction. It does the following:
               1. Assign the restore PassInfo from original PassInfo.
               2. Assign to begin the transaction.
               3. Update the file header.
               4. Call the AfterValidateAndUpload method. This method can be overridden.
               5. Commit the transaction.
      2. In Validate and Upload File Sequential, the application does the following:
         1. It begins the transaction.
         2. It calls the IsFileValid method. This can be overridden if you want it to do any specific task.
         3. It checks if the file is valid. If valid, it will read the record, create a loop, and validate and upload the record. If not valid, it will increment the error count.
         4. If the file is valid and error count is zero, it sets header status to Upload. Otherwise, it sets header status to Review.
         5. It updates the File Header.
         6. It calls the AfterRecordsValidation method, which can be overridden.
         7. It commits the transaction.
2. The application processes the file.
   1. The application calls the InitializeFile method. This is defined in the framework, and you can override it if you want it to do any specific task.
   2. The application checks if the HeaderGroupArray is null and HeaderGroupPosition > 0. If true, it initializes the HeaderGroup position array.
   3. The application checks if Upload to File Detail or Receive to File Detail are true.
      1. If true, it checks the Grouping and Process in Parallel settings.
         1. If Grouping is Yes and Process in Parallel is Yes, the application processes groups from the File Detail in parallel.
            1. The application gets the group value and count from the File Detail. This is defined in entFramework.GetGroupsFromFileDtl.
            2. The application creates the dictionary of HeaderGroup and count.
            3. The application calls a loop. For parallel loop on each group in the dictionary, it gets the new PassInfo and Connection. It then gets the group detail using a query. This is defined in entFramework.GetDetailsForGroup.
               1. It opens another loop and for each record in the group detail records it calls CreatesRecordParams.
               2. It calls the NewHeaderGroup method, which can be overridden.
               3. It calls the ProcessRecord method.
                  1. It checks if the record is empty. If yes, it will return to the calling method.
                  2. It populates the list of fields if it is a delimited file.
                  3. It calls the SetTransactionCode method, which can be overridden.
                  4. It checks for IgnoreRecord. If it is to be ignored (value is Yes), it will return to the calling method.
                  5. For every record in the file, it will check for unsafe characters. If it finds any, it calls the AllowUnsafeRecord method, which can be overridden. It then checks if it is allowed. If false, it raises an exceptio of "Unsafe characters found in the file record." Otherwise, it uses the modified record.
                  6. It gets the record layout. If it can't get the record layout, it calls the IgnoreErrorOnUpload method, which can be overridden and raises the 5006 message.
                  7. It checks if the summary object is not created and the record type is summary. If this returns true, it calls the NewSummary method, which can be overridden. It will only call this method if sfwBindToObjectOnReceive is set at transaction level in S3. Next, it gets the field list for fixed length. It checks for Fixed Length if the record length doesn't match. If this returns true, it calls the IgnoreErrorOnUpload method, which can be overridden and raises the 5001 message.
                  8. It checks the error count. If it is 0, it will bind to the object.
                     1. For the record type, it checks header and summary. If none, it calls the NewDetail method, which can be overridden. This creates the new object based on the object ID mapping in the XML file.
                     2. For each field mapping, it calls the BeforeFieldAssigned method, which can be overridden. It tries to set the value based on mapping. If there is an error, it calls the ContinueOnValueError method on the business object. You can override this, but it has to return one of the following:
                        1. SkipRecord (utlOnRecordError.ContinueSkipRecord): The record is skipped and the application starts processing the next record after adding the error to sgs_file_dtl. If the skip record condition is true, then the status is assigned as Ignore. If the condition is false, the status is assigned as Review.
                        2. WithRecord (utlOnRecordError.ContinueWithRecord): The error is added to sgs_file_dtl_error. It assigns the property based on the modified value.
                        3. RejectFile (utlOnRecordError.RejectFile): It adds the error message and rejects the file.
                     3. As part of the process on upload, it overrides the IgnoreErrorOnUpload method.
                        1. If the error count is 0, it will call the BeforeProcessDetail method, which can be overridden.
                        2. It will check if the ProcessDetail (sfwProcessDetail) S3 flag is set. If yes, it checks if the record type is summary. If yes, it calls the ProcessSummary method, which can be overridden. If no, it calls the ProcessDetail method.
                           1. If it overrides ProcessSummary, it checks if the busSummaryObject exists. If yes, it calls the ProcessHeader method. If no, it calls the BeforeProcessHeader method, which can be overridden. This passes the summary object.
                           2. If it calls ProcessDetail, it sets the page mode based on the detail object's primary key. It calls all the following methods, all of which can be overridden:
                              1. busDetail.BeforeValidate
                              2. busDetail.ValidateHardError: If this finds errors, the application returns to the calling method.
                              3. busDetail.BeforePersistChanges
                              4. busDetail.PersistChanges
                              5. busDetail.ValidateSoftError
                              6. busDetail.ValidateChecklist
                              7. busDetail.UpdateValidateStatus
                              8. busDetail.AfterPersistChanges
                           3. It checks whether the record is rejected. If the record is rejected, it sets the reject flag and inserts a break and exits the loop. If the record is not rejected, it updates the file detail status.
                           4. It checks if the parallel loop on each group in the dictionary is rejected. If no, it calls the ProcessHeaderGroup method, which can be overridden. This method does the following:
                              1. Checks if the group value is summary. If yes, it processes the summary and returns a value of true. In the Process Summary, it checks if busSummaryObject exists. If yes, it calls the ProcessHeader method (this passes the summary object). This method can be overridden. If the ProcessHeader method is overridden, it does the following:
                                 1. it calls the BeforeProcessHeader method, which can be overridden.
                                 2. It sets the page mode of the passed object based on the primary key.
                                 3. It calls the objBusHeader.BeforeValidate method, which can be overridden.
                                 4. It calls the objBusHeader.ValidateHardErrors method, which can be overridden.
                                 5. It calls the objBusHeader.AfterValidate method, which can be overridden.
                                 6. It calls for a loop and checks for each bus (ibusBase) that has changed on the objBusHeader:
                                    1.  it calls the method ibusBase.BeforeValidate, which can be overridden.
                                    2. It calls the method ibusBase.ValidateHardErrors, which can be overridden.
                                    3. It calls the method ibusAfterValidate, which can be overridden.
                                 7. It checks for any error in header validation or loop validation. If yes, it returns to the calling method.
                                 8. It calls the method objBusHeader.BeforePersistChanges, which can be overridden.
                                 9. It checks if it has to skip the soft error validation (objBusHeader.iblnSkipValidateSoftError). If no, it calls objBusHeader.ValidateSoftErrors, objBusHeader.ValidateChecklist, and objBusHeader.UpdateValidateStatus, in that order.
                                 10. It calls for a loop and checks for each bus object (ibusBase) that has changes on objBusHeader if it has skipped the soft error validation (ibusBase.iblnSkipValidateSoftError). If no, then it calls ibusBase.ValidateSoftError, ibusBase.ValidateChecklist, and ibusBase.UpdateValidateStatus, in that order. All these can be overridden.
                                 11. It calls the objBusHeader.AfterPersistChanges method, which can be overridden.
                              2. It checks for any hard errors (hard error or detail errors). If it finds any, it calls ContinueOnHeaderHardError. You can override this, but it has to return one of the following:
                                 1. ContinueSkipRecord: It creates a loop for each record in the group:
                                    1. In the loop it creates file details if none exist.
                                    2. It sets the status of each file detail to Ignored.
                                    3. It checks if the detail record has errors. If yes, it adds all errors to the file details.
                                 2. ContinueWithRecord: It sets flags to reject the file.
                                 3. RejectFile: It sets flags to reject the file.
            4. The application calls UpdatesFileHeaderOnProcess.
         2. If Grouping is Yes and Process in Parallel is No, the application processes groups from the File Detail sequentially.
            1. The application gets the group value and count (entFrameworkGetGroupsFromFileDtl).
            2. The application begins the transaction.
            3. The application creates a dictionary of HeaderGroup and count.
            4. The application creates a loop on each group in the dictionary.
               1. It gets the group using the query entFramework.GetDetailsForGroup: select * from sgs_file_dtl with(nolock) where file_hdr_id=@file_hdr_id and header_group_value=@header_group_value order by line no
               2. It calls for another loop and for each record in the group detail records it calls CreatesRecordParams.
               3. It calls the NewHeaderGroup method, which can be overridden.
               4. It processes the record.
                  1. If the record is rejected, it sets the rejected flag and inserts a break and exits the loop.
                  2. If the record is not rejected, it calls UpdateFileDetailStatus.
               5. It checks each group in the dictionary for rejection.
                  1. If the group is rejected, it breaks and exits the loop.
                  2. It the group is not rejected, it calls ProcessHeaderGroup, which can be overridden.
               6. It checks the commit single flag in the XML file. If the flag is yes, it will update the line number on the file header, commit the transaction, and begin the transaction.
            5. The application updates the file header on process.
            6. The application commits the transaction.
         3. If Grouping is No and Process in Parallel is Yes, the application processes groups from the File Detail in parallel.
            1. The application creates a list of file detail IDs (entFramework.GetDetailIDsForProcessing).
            2. The application creates a parallel loop on each File Detail ID:
               1. It gets new PassInfo and Connection.
               2. It begins the transaction.
               3. It loads the file detail using the File Detail ID.
               4. It calls CreateRecordParams.
               5. It processes the record.
               6. It calls UpdateFileDetailStatus.
               7. It checks if the record is rejected:
                  1. If rejected, it sets the reject flag, rolls back the transaction, and inserts a break and exits the loop.
                  2. If not rejected, it commits the transaction.
            3. The application calls UpdateFileHeaderOnProcess.
         4. If Grouping is No and Process in Parallel is No, the application processes groups from the File Detail sequentially.
            1. The application begins the transaction.
            2. The application creates a loop on each detail record using entFramework.GetUnprocessedDetailsForHeader.
               1. It loads the file detail using the DataRow.
               2. It calls CreateRecordParams.
               3. It processes the record.
               4. It calls UpdateFileDetailStatus.
               5. It checks if the record is rejected:
                  1. If rejected, it sets the reject flag and inserts a break and exits the loop.
                  2. If not rejected, it checks CommitSingle. If yes, it updates the processed line number to the file header. It commits the transaction and begins the transaction.
               6. It calls UpdateFileHeaderOnProcess.
               7. It commits the transaction.
      2. If false, it checks the Grouping and Process in Parallel settings.
         1. If Grouping is Yes and Process in Parallel is Yes, the framework has no contingency for processing in parallel.
         2. If Grouping is Yes and Process in Parallel is No, the application processes groups from the File Detail sequentially.
            1. The application gets the database of the ignored lines for a file (entFramework.GetIgnoredFileDetailsByHeaderId).
            2. The application begins the transaction.
            3. The application creates a loop of each record in the file.
               1. It checks if the record is blank. If so, it continues to the next record.
               2. It checks if the line number is in the ignored data table. If so, it continues to the next record.
               3. It calls CreateRecordParams.
               4. It checks if the record is delimited by a character. If so, it populates the field list.
               5. It calls the SetTransactionCode method, which can be overridden.
               6. It checks if the record is fixed length. If so, it populates the field list based on the record layout.
               7. It calls the SetHeaderGroupValue method, which can be overridden.
               8. It checks if the group values changed. If so, it checks if the old group value is empty. If so, it calls the NewHeaderGroup method, which can be overridden. If not, it calls the ProcessHeaderGroup method, which can be overridden. This method does the following:
                  1. It checks whether it has itself succeeded. If not, it will set a reject flag.
                  2. It checks for rejection. If yes, it inserts a break and exits the loop.
                  3. It checks if it is to commit single. If so, it updates the processed line number in the file header, commits the transaction, and begins the transaction.
               9. It processes the record.
               10. It checks whether the record is rejected.
                   1. If rejected, it sets a reject flag, rolls back the transaction, and inserts a break and exits the loop.
                   2. If not rejected, it commits the transaction.
               11. It updates the file detail status. The loop ends.
            4. The application checks if it is rejected.
               1. If it is not rejected, it calls the ProcessHeaderGroup method, which can be overridden. If ProcessHeaderGroup is not successful, it sets a reject flag.
            5. It calls the UpdateFileHeaderOnProcess method.
         3. If Grouping is No and Process in Parallel is Yes, the application processes groups from the File Detail in parallel.
            1. The application gets the data table of the ignored lines for the file (entFramework.GetIgnoredFileDetailsByFileHeaderId).
            2. The application calls a parallel loop on each record in the file.
               1. It checks if the record is blank. If so, it continues to the next record.
               2. It gets new PassInfo and Connection.
               3. It begins the transaction.
               4. It checks if the line number is in the ignored data table. If so, it continues to the next record.
               5. It calls CreateRecordParams.
               6. It processes the record.
               7. It calls UpdateFileDetailStatus.
               8. It checks if the record is rejected.
                  1. If rejected, it sets the reject flag, rolls back the transaction, and inserts a break and exits the loop.
                  2. If not rejected, it commits the transaction.
            3. The application calls UpdateFileHeaderOnProcess.
         4. If Grouping is No and Process in Parallel is No, the application processes groups from the File Detail sequentially.
            1. The application gets the data table of the ignored lines for the file (entFramework.GetIgnoredFileDetailsByFileHeaderId).
            2. The application begins the transaction.
            3. The application calls a loop on each record in the file.
               1. It checks if the record is blank. If so, it continues to the next record.
               2. It checks if the line number is in the ignored data table. If so, it continues to the next record.
               3. It calls CreateRecordParams.
               4. It calls ProcessRecord.
               5. It calls UpdateFileDetailStatus.
               6. It checks if the record is rejected.
                  1. If rejected, it sets the reject flag, rolls back the transaction, and inserts a break and exits the loop.
                  2. If not rejected, it checks if it has to commit single. If so, it updates the process line number, commits the transaction, and begins the transaction.
            4. The application calls UpdateFileHeaderOnProcess.
   4. The application calls the FinalizeFile method, which can be overridden.

This post is part of the Files topic. Click here to open the Files Overview.