Start and Monitor The Workflow

1. If the Workflow is valid, it is ready for execution. In the Workflow Manager, use one of the following methods to start the wf_Employee_Name_x Workflow. Please use the workflow created in the exercise Understand Informatica PowerCenter Workflow Designer

1. Select WORKFLOWS | START WORKFLOW.
2. Right-click in the Workspace and select Start Workflow or
3. Right-click on the wf_Employee_Name_x Workflow in the Navigator Window and select Start Workflow.

2. To monitor a Workflow the Workflow Monitor must be opened. This is opened automatically when a workflow is executed. If this does not happen perform the following steps to start the Workflow Monitor

1. Select Start | Programs | Informatica PowerCenter Client | Workflow Monitor.

• To connect to Repository use one of the following methods:
• In the Workflow Monitor Select REPOSITORY | CONNECT or
• Click on the icon in the toolbar; or double-click on Repository in the Navigator Window. The Connect To Repository box appears.

Note: Enter repository, server and login details.

3. Right click on the server name and select Connect.
4. Double click on the folder to view Workflow sessions.
5. You get two views, the Gantt Chart View and the Task View.

6. Select the Gantt Chart tab. This view displays details about workflow runs in chronological format. It displays the following information as shown below.

7. Select the Task View. This view displays details about workflow runs in a report format. The Status column gives the following information.

1. A Succeeded status if the PowerCenter Server was able to successfully complete a Workflow or Task.
2. A Failed status may occur if the PowerCenter Server failed the Workflow or Task due to fatal processing errors.
3. A Running status if the PowerCenter Server is still processing is still processing Workflow or Task.

8. View the Session properties by doing one of the following.

1. Right click on the Session selected and select Properties; or
2. Click on the Session Properties icon.

9. The Properties tab of the S_Employee_Name_x dialog box opens. The Session should display the number of Target Success Rows as shown below.

10. Click on the Transformation Statistics tab. More detail on the number of rows handled by the Server are shown here.

1. Applied rows are rows the Informatica Server successfully produced and applied to the target without errors.
2. Affected rows are generated by the Server and ‘affected to’ (or accepted by) the target.
3. Rejected rows are either those read rows that the Server dropped during the transformation process, or, the rows that were rejected when writing to the target.

11. View Session Log to determine what occurred during the system run. To view detailed Session information, do one of the following

1. Right-click on the Session in the Name column and select Get Session Log.
2. Select the Session name and click on the icon.

Video Tutorial.

• Repository Name : Your Repository Name.
• Description :  An optional description about the repository.
• Location : Choose the Domain you have already created. If you have only one Domain, this value will be pre populated.
• License : Choose the license key from the drop down list.
• Node : Choose the node name from the drop down list.

• Database Type : Choose your Repository database (Oracle/SQL Server/Sybase)
• Username : Database user ID to connect database.
• Password : Database user Password.
• Connection String : Database Connection String.
• Code Page : Database Code Page
• Table Space : Database Table Space Name
• Choose “No content exists under specified connection string. Create new content”
• Choose “Enable version control”

• Operation Mode : Normal
• Security Audit Trail : No

Framework Components

1. A Relational Table :- To store the meta data.
2. Mapping, Workflow variables : Variables to store and process latest timestamp of processed records.
3. A Reusable Expression :- A reusable expression transformation to find latest timestamp of processed records.
4. Pre, Post Session Command Task :- Command task to collect the meta data.
5. Reusable Worklet :- Worklet to log the data load details into the relational table.

1. Relational Table

• ETL_JOB_NAME : ETL job name or Session name.
• ETL_RUN_DATE : ETL job execution date.
• DATA_START_TIME : Least timestamp of processed records
• DATA_END_TIME : Latest timestamp of processed records
• SRC_TABLE : Source table used in the ETL job.
• TGT_TABLE : Target table used in the ETL job.
• ETL_START_TIME : ETL job execution start timestamp.
• ETL_END_TIME : ETL job execution end timestamp.
• SRC_RECORD_COUNT : Number of records read from source.
• INS_RECORD_COUNT : Number of records inserted into target.
• UPD_RECORD_COUNT : Number of records updated in target.
• ERR_RECORD_COUNT : Number of records error out in target.
• ETL_STATUS : ETL Job status, SUCCESS or FAILURE.
• ETL_CREATE_TIME : Record create timestamp.
• ETL_UPDATE_TIME : Record update timestamp.

2. Mapping and Workflow Variables

3. Reusable Expression

• SETMINVARIABLE($$M_DATA_START_TIME,DATA_TIME)
• SETMAXVARIABLE($$M_DATA_END_TIME,DATA_TIME)

4. Pre and Post Session Command Task

• ETL Start time
• ETL End time
• ETL Job name
• Source table name
• Target table name
• Source record count
• Records inserted count
• Records updated count
• Error record count
• ETL Job status

• $PMSessionName : Name of the Informatica session.
• $PMSourceName@TableName : Name of the source table name.
• $PMTargetName@TableName : Name of the source table name.
• $PMSourceQualifierName@numAffectedRows : Number of records returned from source.
• $PMTargetName@numAffectedRows : Number of record inserted/updated into the target table.
• $PMTargetName@numRejectedRows : Number of records error out in target.

Pre Session Command Task

echo %DATE:~10,4%-%DATE:~4,2%-%DATE:~7,2% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2%,
> $PMSourceFileDir\$PMWorkflowName_stat.txt

Post Session Success Command Task

echo %DATE:~10,4%-%DATE:~4,2%-%DATE:~7,2% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2%,
$PMSessionName,
$PMSTG_CUSTOMER_MASTER@TableName,
$PMINS_CUSTOMER_MASTER@TableName,
$PMSQ_STG_CUSTOMER_MASTER@numAffectedRows,
$PMINS_CUSTOMER_MASTER@numAffectedRows,
$PMUPD_CUSTOMER_MASTER@numAffectedRows,
$PMINS_CUSTOMER_MASTER@numRejectedRows,
SUCCESS,
>> $PMSourceFileDir\$PMWorkflowName_stat.txt

Post Session Failure Command Task

echo %DATE:~10,4%-%DATE:~4,2%-%DATE:~7,2% %TIME:~0,2%:%TIME:~3,2%:%TIME:~6,2%,
$PMSessionName,
$PMSTG_CUSTOMER_MASTER@TableName,
$PMINS_CUSTOMER_MASTER@TableName,
$PMSQ_STG_CUSTOMER_MASTER@numAffectedRows,
$PMINS_CUSTOMER_MASTER@numAffectedRows,
$PMUPD_CUSTOMER_MASTER@numAffectedRows,
$PMINS_CUSTOMER_MASTER@numRejectedRows,
FAILURE,
>> $PMSourceFileDir\$PMWorkflowName_stat.txt

• Pre, Post session commands need to be changed based on Informatica server operating system.
• Highlighted part of the script need to be change based on the source, target table instance name used in the mapping.

5. Reusable Worklet

Reusable Mapping

A reusable mapping will be created to read data from the comma delimited file generated by the pre and post session command task.

This mapping takes two input parameters, Create the mapping and add two mapping variables in the mapping as shown below.

• $$M_DATA_START_TIME as Date/Time
• $$M_DATA_END_TIME as Date/Time

This mapping reads data from the file generated by the Pre, Post session command task. Mapping will include an expression transformation to generate the data elements required in the target table, with below OUTPUT ports, This expression transformation takes two input ports from the source file.

• ETL_RUN_DATE :- TRUNC(SESSSTARTTIME)
• DATA_START_TIME :- $$M_DATA_START_TIME
• DATA_END_TIME :- $$M_DATA_END_TIME
• ETL_CREATE_TIME :- SESSSTARTTIME
• ETL_UPDATE_TIME :- SESSSTARTTIME
• O_ETL_START_TIME :- TO_DATE(LTRIM(RTRIM(ETL_START_TIME)),'YYYY-MM-DD HH24:MI:SS')
• O_ETL_END_TIME :- TO_DATE(LTRIM(RTRIM(ETL_END_TIME)),'YYYY-MM-DD HH24:MI:SS')

Below is the complete mapping structure, created to populate target table 'ETL_PROCESS_STAT' 

Reusable Worklet

Reusable worklet is created based on the mapping created in the last step. Create the worklet and add two worklet variables.

• $$WL_DATA_START_TIME as Date/Time
• $$WL_DATA_END_TIME as Date/Time

Now create the session in the worklet, which will be configured to read data from the file created by the pre, post session command as shown below. This session is based on the reusable mapping created in the previous step.

Note : Make sure the Source File Directory and Source File name are given correctly based on the file generated by pre/post session command

Assign the worklet variables to the mapping variables as shown below, using the pre-session variable assignment option in the components tab.

With that we are done with the configuration required for the worklet.

Framework implementation in a workflow

Now lets see how we implement the Change Data Capture Frame work in a mapping.

Mapping

Lets start the mapping creation and add two mapping variables as shown below.

• $$M_DATA_START_TIME as Date/Time , Initial Value 12/31/2050 00:00:00.000000000
• $$M_DATA_END_TIME as Date/Time , Initial Value 12/31/2050 00:00:00.000000000

Note : Give the initial value for both the variables

Add source and source qualifier to the designer work space, open the source qualifier and give the filter condition to get the latest data from the source.

• STG_CUSTOMER_MASTER.UPDATE_TS > CONVERT(DATETIME,'$$M_DATA_END_TIME')

Hint : Use the column in the filter condition, based on which the Change Data Capture is built up on.

Add the Reusable Transformation 'EXP_CDC_TIMESTAMP' to the mapping and map the column 'UPDATE_TS'  from the source qualifier to the input port of the expression.

Hint : Use the column from the source qualifier, based on which Change Data Capture is built on. 

Note : The reusable transformation will find the least and latest timestamp and will store in the mapping variables, which can be used in the subsequent runs.

Map the DUMMY column from 'EXP_CDC_TIMESTAMP' to the down stream transformation and complete all other transformation required in the mapping. 

Workflow

Once the mapping is complete. lets create the workflow and add two workflow variables as shown below.

• $$WF_DATA_START_TIME as Date/Time
• $$WF_DATA_END_TIME as Date/Time

Create the session in the workflow and add the Pre, Post session commands, which creates the flat file with the session run details.

Now map the mapping variables to the workflow variables as below.

Add the worklet to the workflow and assign the workflow variables to the worklet variable as in below image.

With that we are done with the configuration. And below is the structure of the completed workflow, with Change Data Capture Framework.

Hope you enjoyed this post.  We will be expanding this framework by adding features like notification capability, detail error capturing, change data capture etc... in the coming posts. Please leave your comments and thought about this.

Change Data Capture Implementation

• SETMAXVARIABLE($M_DATA_END_TIME,UPDATE_TS)

That’s all you need to configure Change Data Capture, Now create your workflow and run the workflow. 

Once you look into the session log file you can see the mapping variable value is retrieved from the repository and used in the source SQL, just like shown in the image below.

Hope you enjoyed this article. Please let us know your comment and feed back.

Informatica Functions Used

Note : Use the ERROR, ABORT function for both input and output port default values. You might use these functions for input ports to keep null values from passing into a transformation and use for output ports to handle any kind of transformation error.

Informatica Implementation

For the demonstration lets consider a workflow which loads daily credit card transactions and below two user defined data validation checks

1. Should not load any transaction with 0 (zero) amount, but capture such transactions into error tables
2. Should not process any transactions with out credit card number and Stop the workflow.

Mapping Level Changes

To handle both the exceptions, lets create an expression transformation and add two variable ports.

• TEST_TRANS_AMOUNT as Variable Port
• TEST_CREDIT_CARD_NB as Variable Port

Add below expression for both ports. First expression will take care of the user defined data validation check No 1 and second expression will take care of user defined data validation check No 2.

• TEST_TRANS_AMOUNT :- IIF(TRANS_AMOUNT = 0,ERROR('0 (Zero) Transaction Amount'))
• TEST_CREDIT_CARD_NB :- IIF(ISNULL(LTRIM(RTRIM(CREDIT_CARD_ND))),ABORT('Empty Credit Card Number'))

The complete expression transformation is shown in below image.

Now insert this transformation in the mapping where you need the data validation and complete the mapping.

Hint :  This Expression can be converted into a Reusable transformation, So that any mapping needs this data validation can reuse this transformation.

Session Level Changes

Once the mapping is complete, configure the session and provide the settings for row error logging as shown in below image. Please read the article Error handling made easy using Informatica Row Error Logging for more details on row error logging.

With this configuration we specified, Informatica PowerCenter will create four different tables for error logging and the table details as below.

• ETL_PMERR_DATA :- Stores data about a transformation row error and its corresponding source row.
• ETL_PMERR_MSG :- Stores metadata about an error and the error message.
• ETL_PMERR_SESS :- Stores metadata about the session.
• ETL_PMERR_TRANS :- Stores metadata about the source and transformation ports, when error occurs.

With this, we are done with the setting required to capture user defined errors.  Any data records which violates our data validation check will be captured into PMERR tables mentioned above.

Report the Error Data.

Now we have the error data stored in the error table, we can pull the error report using an SQL querry. Below is a basic query to get the error report. We can be more fancy with the SQL and get more information from the error tables.

select 

       sess.FOLDER_NAME as 'Folder Name',

       sess.WORKFLOW_NAME as 'WorkFlow Name',

       sess.TASK_INST_PATH as 'Session Name',

       data.SOURCE_ROW_DATA as 'Source Data',

       msg.ERROR_MSG as 'Error MSG'
from 

       ETL_PMERR_SESS sess
left outer join ETL_PMERR_DATA data
      on data.WORKFLOW_RUN_ID = sess.WORKFLOW_RUN_ID and 

      data.SESS_INST_ID = sess.SESS_INST_ID
left outer join ETL_PMERR_MSG msg
      on msg.WORKFLOW_RUN_ID = sess.WORKFLOW_RUN_ID and 

      msg.SESS_INST_ID = sess.SESS_INST_ID
where 

      sess.FOLDER_NAME = <Project Folder Name> and
      sess.WORKFLOW_NAME = <Workflow Name> and
      sess.TASK_INST_PATH = <Session Name> and

      sess.SESS_START_TIME = <Session Run Time>

Pros and Cons of this Approach.

We should know the Pros and Cons of this approach  before applying this to your project.

Pros.

1. Out of the box Solution Provided by Informatica.
2. Less Coding and Testing efforts required by the development team.

Cons.

1. Added overhead to the Session performance, which is expected and acceptable.

Hope you enjoyed this article. Please leave us a comment below, if you have any difficulties implementing this error handling approach. We will be more than happy to help you.

• To get a summary of sales by item description, state and month,
• Collect data from various relational tables sources to consolidate the information.
• Create a relational target containing the summary wise details are created.

I. Start the Designer

II. Create and verify source definitions

1. Select SOURCES | IMPORT FROM DATABASE to import the ITEMS, ORDERS, ORDER_ITEMS and STORES tables.  Hint: Press the Ctrl key while selecting each table with a single mouse click in the Import Tables box.
2. In the Source Analyzer workbook , expand the Key Types column for each source definition.
3. Verify primary/foreign key relationships by. 

• Verifying Key Types.
• Observing the key relationships indicated by link lines between the tables, shown below.

III. Edit source definitions

1. In the Description window of the ORDERS source definition, enter: “This is the ORDERS source table containing all of the orders for the company”
2. Click in the Column Name column on the ORDER_ID line and enter In the Description window : “This is the order number uniquely distinguishing one order from another.”
3. Save your work.

IV. Design a Target Schema

1. Switch to Target Designer.
2. Create a target schema from scratch and name it Tgt_SalesSummary_x. Your target should look like in the figure shown below.
3. Create the physical table in the database so you can load data. Select the options as shown below Hint: Select TARGETS | GENERATE/EXECUTE SQL
4. Click on Edit SQL file to view the script file created.
5. Save the newly designed schema to the repository.

V. Drag sources and create Source Qualifier Transformation

1. Switch to Mapping Designer.
2. In Designer’s Navigator Window, select your folder.
3. Create a new mapping.
4. Enter M_SalesSummary_x for the new mapping name
5. Disable automatic creation of Source Qualifier transformation. Hint : In TOOLS | OPTIONS, click on the Format tab. 
6. In Designer’s Navigator Window, expand the Sources section (node) in the Navigator Window, select the ITEMS, ORDER_ITEMS, ORDERS, and STORES tables and drag them to the far left side of the workspace.
7. Create the Source Qualifier transformation .
8. Enter the name of the transformation and click on create. Select the sources in the Select Sources for Source Qualifier Transformation. Click on OK and Done.
   Note: Blue link lines appear from the source definitions to the new Source. All columns (ports) from each of the four source definitions are linked into the new Source Qualifier.9. Rename the transformation to SQ_SalesSummary_x.

VI. Drag target into the Workspace

1. Select the Tgt_SalesSummary_x table and drag it to the far right side of the workbook.

VII. Create the Expression transformation 

1. Create the  expression transformation and place it to the right of the Source Qualifier transformation.
2. Link the following ports from the SQ_SalesSummary_x to the new Expression transformation: ITEM_DESC, PRICE,  QUANTITY,  DATE_ENTERED, STATE
   Hint: Select the Link Columns icon in the toolbar .
3. Rename the transformation to Exp_SalesSummary_x

VIII. Use functions in the Expression transformation

1. Click on the Ports tab.
2. Disable the output port of DATE_ENTERED column by removing the check mark in the options box in the ‘O’ column. This will make the port an input-only port.
3. Add a new port and name it MONTH. 
   Hint : Selecting the STATE port before clicking the icon will add the new port immediately after STATE.
4. Disable the input port to MONTH by removing the check mark in the options box in the ‘I’ column. The Expression section of MONTH becomes eligible for editing.
5. Add a new port Year.
6. Let Year be an output port.
7. On the line for MONTH, click on the downward arrow to the far right of the Expression column. This opens the Expression Editor dialog box.

IX. Create the expression for the Month Port

1. Enter the expression that defines MONTH in the Expression Editor.
2. Delete the text MONTH.
3. Select the Functions tab.
4. Click on the ‘+’ next to Date to open the Date folder. A list of all date functions appears.
5. Double-click the To_Char() function. The To_Char() function appears in the Formula Window.
6. To define the port from which this function will extract the value for MONTH,select the Ports tab. All ports from all transformations in the mappings appear If you click on the ‘+’ next to each transformation
7. Position the mouse in the parenthesis and double click the DATE_ENTERED port for the Expression transformation. DATE_ENTERED now appears within the expression you are building in the Formula: window, within the parentheses for the To_Char() function.
8. Use the Keypad (below the Formula: window) to add a comma to the expression, after DATE_ENTERED.
9. Then type ‘Month’. Include the single quotes.
10. The expression is now complete. The finished expression should read: TO_CHAR(DATE_ENTERED, ‘Month’)
11. Click Validate to parse the expression.
12. After the expression has been parsed successfully, click OK to exit the Expression Editor.

X. Create the expression for Year

1. Configure the Year port in a similar fashion as the MONTH port, entering the expression : TO_CHAR(DATE_ENTERED’ ‘YYYY’)
2. Your Ports tab will look something like the following table:
3. Click OK to exit the Edit Transformation dialog box.

XI. Create the Aggregator Transformation to get total price and total quantity

1. Click on the aggregator transformation icon in the toolbar .
2. Click the mouse to the right of Exp_SalesSummary_x. An Aggregator transformation appears.
3. Link the following columns from Exp_SalesSummary_x to the Aggregator transformation: ITEM_DESC, PRICE, QUANTITY, STATE, MONTH, and YEAR Hint : Drag the above columns from Expression transformation into Aggregator transformation. Make sure the link columns icon is selected .
4. Rename the transformation to Agg_SalesSummary_x
5. Click OK.
6. Click on the Ports tab.
7. Disable the output ports for PRICE and QUANTITY. They will now be input-only ports.
8. Add new ports for the TOTAL_QTY and TOTAL_PRICE. They will be output-only ports.
   Note: The values for these ports will be calculated before data leaves the Aggregator.
9. Enter expressions for these two new ports in the Expression Editor. TOTAL_QTY : SUM(QUANTITY)
   TOTAL_PRICE: SUM(QUANTITY * PRICE)
   Hint : The SUM function is found in the Aggregate folder. It is not available for use in expressions in any transformation except the Aggregator transformation.
10. Validate the expressions.
11. Check the GroupBy boxes on the lines for ITEM_DESC, STATE, MONTH, and YEAR These are the columns by which we want to summarize. Your Ports tab will look something like the table below:
    Note : The order of GroupBy ports should be in the sequence as shown. Select the and to move the column up and down.
12. Click OK to exit the Edit Transformation dialog box.
13. Link the following ports from Agg_SalesSummary_x to Tgt_SalesSummary_x
    ITEM_DESCRIPTION ->DESCRIPTION
    TOTAL_QTY –> TOTAL_QTY
    TOTAL_PRICE –> TOTAL_PRICE
    STATE –> STATE
    MONTH –> MONTH
    YEAR –> YEAR
14. Save changes to the repository.
15. Review the information on Designer’s Output window.
16. As the repository is saved, the Output window will display status information relevant to the metadata you have entered into the repository.
17. If the mapping is invalid, make changes and validate the mapping again till valid.
    Note: To save the cached output information in the Output window, select the tab so that the Output window shows messages related to the tab and select REPOSITORY | SAVE OUTPUT AS.
18. The Final mapping will look like the one given below.
    

XII. Create a Workflow and Session Task

1. Start the Workflow Manager, connect to the repository and open your folder.
2. Create a Workflow. Enter the workflow name as wf_SalesSummary_x.
   
3. There are several parameters in the new workflow to be set. Under the Properties tab, note Attribute 2 displays the name of the Workflow Log File: wf_SalesSummary_x.log
4. Add a session task
   1. Create a Session task and name it s_SalesSummary_x task
   2. Select the M_SalesSummary_x mapping from the list of valid mappings and click on OK.
   3. Enter the following description and double click on the session task to edit the session properties
   4. Enter the description for the session in the General tab.
   5. Under the Properties tab, you can enter session log file name, session log file directory, and other general session settings.
   6. Select the Source Database Connection
   7. Select the Target Database Connection
      
   8. Select the Mapping tab and click on the Targets folder. Under Properties, select the Target load type as Normal.
   9. Select the Transformations tab. This section lets you override individual transformation attributes. Overrides may apply to any property on any transformation used within the mapping.
      
   10. Click OK to close the Edit Tasks dialog box.

XIII. Link Workflow Tasks

1. Link Start_SalesSummary_x and s_SalesSummary_x.
2. Validate the Workflow.
   1. Locate the Validate tab in the Output Window at the bottom of the Workflow Manager and view the results of the Validation checks.
   2. Repeat the validation process until the Workflow is valid.
3. Save changes to the repository.
4. Start the Workflow.

XIV. Monitor a Workflow

1. Open Workflow Monitor.
2. Select the Gantt Chart tab
3. Double-click on the folder to view the previously processed workflows Drill down by double clicking on each object all the way until the session task s_SalesSummary_x, appears.
4. Note the status of the s_SalesSummary_x session task.
5. View the Session properties and check if it displays the number of Target Success Rows as shown below:
6. Click on the Transformation Statistics tab. More detail on the number of rows handled by the Server are shown here:
7. View Session Log and read the messages.

1. Mapplet

If you have several fact tables that require a series of dimension keys, you can create a mapplet containing a series of Lookup transformations to find each dimension key. You can then use the mapplet in each fact table mapping.

2. Reusable Transformation

3. Shared Folder

4. Global Repository

5. Worklet

A worklet is an object created by combining set of tasks to build a workflow logic. Worklet can be reused in multiple  workflows, which can be configured to run concurrently. You can create a worklet in the Worklet Designer.

Reusable worklet implemented in the Change Data Capture Framework, which is discussed in one of our prior article is a practical  application of worklet.

6. Reusable Session

A session is a set of instructions that tells the Integration Service how and when to move data from sources to  targets. You can create a reusable Session task in the Task Developer. A reusable session can be used multiple workflows and even in a worklet.

Reusable session used in the Operational Metadata Logging Framework, which is discussed in one of our prior article is a practical  implementation of reusable session.

7. Reusable Tasks

Apart from the reusable session task, we can create reusable email task and reusable command task in the Task  Developer. We can reuse these reusable tasks in multiple workflows and worklets.

A reusable email task can be used to create a standard session failure email notification, which can be reused in different session tasks.

8. Mapping Parameter

Define values that remain constant throughout a session, such as state sales tax rates. When declared in a mapping or mapplet, $$ParameterName is a user-defined mapping parameter.

9. Mapping Variable

Define values that can change during a session. The Integration Service saves the value of a mapping variable to the repository at the end of each successful session run and uses that value the next time you run the session. When declared in a mapping or mapplet, $$VariableName is a mapping variable.

10. WorkFlow Variable

You can create user defined workflow variables to pass variable values between mapping, sessions within a workflow. Even if the workflow variables alone do not give code reusability, it works with other components to provide reusability.

11. Worklet Variable

User defined worklet variables can be create in worklets to pass values between mapping, sessions, worklets within a workflow. Worklet variables by itself do not give code reusability, but it works with other components to facilitate code reusability.

• Service variables.
• Service process variables.
• Session parameters.
• $Source. $Target connection variables.
• Email variables.
• Local variables.
• Built-in variables.

• Create a mapping which reads from a flat file and creates a relational table consisting of new customers
• Analyze a fixed width flat file
• Configure a Connected Lookup transformation
• Use a Filter transformation to exclude records from the pipeline.

I. Connect to the Repository

1. Connect to the repository.
2. Open the folder where you need the mapping built.

II. Analyze the source files

1. Import the flat file definition (say Nielsen.dat) into the repository.
2. Select SOURCES | IMPORT FROM FILE from the menu.
3. Select Nielsen.dat from the source file directory path. Hint : Be sure to set the Files of type: to All files (*.*) from the pull-down list, before clicking on OK.
   1. Set the following options in the Flat File Wizard:
   2. Select Fixed Width and check the Import field names from first line box. This option will extract the field names from the first record in the file.
   3. Create a break line or separator between the fields.
   4. Click on NEXT to continue.
   5. Refer Appendix A to see the structure of NIELSEN.DAT flat file.
4. Change field name St to State and Code to Postal_Code. Note : The physical data file will be present on the Server. At runtime, when the Server is ready to process the data (which is now defined by this new source definition called Nielsen.dat) it will look for the flat file that contains the data in Nielsen.dat.
5. Click Finish.
6. Name the new source definition NIELSEN. This is the name that will appear as metadata in the repository, for the source definition.

III. Design the Target Schema

1. Switch to Target Designer.
2. Select EDIT | CLEAR if necessary to clear the workspace. Any objects you clear from the workspace will still be available for use in Designer’s Navigator Window, in the Targets node.
3. Drag the NIELSEN source definition from the Navigator Window into the workspace to automatically create a target table definition. You have just created a target definition based on the structure of the source file definition. You now need to edit the target table definition.
4. Rename the table as Tgt_New_Cust_x.
5. Enter the field names as mentioned in the Figure below .Change the Key Type for Customer_ID to Primary Key. The Not Null option will automatically be checked. Save the repository.
6. The target table definition should look like this
7. Create the physical table in the Oracle Database so that you can load data. Hint : From the Edit table properties in Target designer, change the database type to Oracle.

IV. Create the mapping and drag the Source and Target

1. Create a new mapping with the name M_New_Customer_x
2. Drag the source into the Mapping Designer workspace. The SourceQualifier should be automatically created.
3. Rename the Source Qualifier as SQ_NIELSEN_x
4. Drag the target (Tgt_New_Cust_x) into the Mapping Designer workspace

V. Create a Lookup Transformation

1. Select TRANSFORMATION | CREATE.
2. Select Lookup from the pull-down list.
3. Name the new Lookup transformation Lkp_New_Customer_x.
4. You need to identify the Lookup table in the Lookup transformation. Use the CUSTOMERS table from the source database to serve as the Lookup table and import it from the database.
5. Select Import to import the Lookup table.
6. Enter the ODBC Data Source, Username, Owner name, and Password for the Source Database and Connect.
7. In the Select Tables box, expand the owner name until you see a TABLES listing.
8. Select the CUSTOMERS table.
9. Click OK.
10. Click Done to close the Create Transformation dialog box. Note : All the columns from the CUSTOMERS table are seen in the transformation.
11. Create an input-only port in Lkp_New_Customer_x to hold the Customer_Id value, coming from SQ_NIELSEN_x .
    1. Highlight the Cust_Id column from the SQ_NIELSEN_x
    2. Drag/drop it to Lkp_New_Customer_x.
    3. Double-click on Lkp_New_Customer_x to edit the Lookup transformation.
    4. Click the Ports tab, make Cust_Id an input-only port.
    5. Make CUSTOMER_Id a lookup and output port.
12. Create the condition for lookup.
    1. Click the Condition Tab.
    2. Click on the    icon.
    3. Add the lookup condition: CUSTOMER_ID = Cust_Id.
       Note : Informatica takes its ‘best guess’ at the lookup condition you intend, based on data type and precision of the ports now in the Lookup transformation.
13. Click the Properties tab.
14. At line 6 as shown in the figure below, note the Connection Information.
    

VI. Create a Filter Transformation

1. Create a Filter transformation that will filter through those records that do not match the lookup condition and name it Fil_New_Cust_x.
2. Drag all the ports from Source Qualifier to the new Filter. The next step is to create an input-only port to hold the result of the lookup.
3. Highlight the CUSTOMER_ID port from Lkp_New_Customer_x .
4. Drag it to an empty port in Fil_New_Cust_x .
5. Double-click Fil_New_Cust_x to edit the filter.
6. Click the Properties tab.
7. Enter the filter condition: ISNULL(CUSTOMER_ID). This condition will allow only those records whose value for CUSTOMER_ID is = null, to pass through the filter.
8. Click OK twice to exit the transformation.
9. Link all ports except CUSTOMER_ID from the Filter to the Target table.
   Hint : Select the LAYOUT | AUTOLINK menu options, or right-click in the workspace background, and choose Auto link. In the Auto link box, select the Name radio button. This will link the corresponding columns based on their names.
   
10. Click OK.
11. Save the repository.
12. Check the Output window to verify that the mapping is valid.
13. Given below is the final mapping.

VII.    Create the Workflow and Set Session Tasks Properties

1. Launch the Workflow Manager and connect to the repository.
2. Select your folder.
3. Select WORKFLOWS | CREATE to create a Workflow as wf_New_Customer_x.
4. Select TASKS | CREATE to d create a Session Task as s_New_Customer_x.
5. Select the M_New_Customer_x mapping.
6. Set the following options in the Session Edit Task:
   1. Select the Properties tab. Leave all defaults.
7. Select the Mapping tab.
   1. Select the Source folder. On the right hand side, under Properties, verify the attribute settings are set to the following:
      1. Source Directory path = $PMSourceFileDir\
      2. File Name = Nielsen.dat (Use the same case as that present on the server)
      3. Source Type: Direct
         Note : For the session you are creating, the Server needs the exact path, file name and extension for the file as it resides on the Server, to use at run time
   2. Click on the Set File Properties button.
   3. Click on Advanced. 
      
   4. Check the Line sequential file format check box.
   5. Select the Targets folder. 
      1. Under Connections on the right hand side, Select the value of
         Target Relational Database Connection.
   6. In the Transformations Folder, Select the Lkp_New_Customer transformation.
      1. On the right hand side, in Connections, Select the Relational Database Connection for the Lookup Table. Figure
8. Run the Workflow.
9. Monitor the Workflow.
10. View the Session Details and Session Log.
11. Verify the Results from the target table by running the query SELECT * FROM Tgt_New_Cust_x;

Video Tutorial

Joiner Transformation can be used to achieve the functionality of SQL join Operation including  full outer join. Additionally we can use Joiner to join data from heterogeneous data sources. But it is limited with the operators, which can be used in the join condition, it can use only equal (=) condition in the join. In this article lets see how we can unlock this limitation using Informatica PowerCenter Active LookUp transformation.

To overcome this disability we will be using Active LookUp Transformation, which is available from Informatica PowerCenter Version 9x.

What is Active LookUp

From Informatica PowerCenter Version 9x onwards we can configure the lookup transformation to return all the rows from the lookup table matching the lookup condition. This Lookup transformation becomes an active transformation. For the Active LookUp 'Lookup Policy on Multiple Match' property will be 'Use All Values'. This property becomes read-only and cannot be changed after the transformation is created.

How to Configure Active LookUp

Just like any other transformation start creating the transformation.

Choose the LookUp Table from the popup window and select 'Return All Values on Multiple Match'. This property will set the lookup as active lookup transformation.

From the properties tab you can see the 'Lookup Policy on Multiple Match' property is set as 'Use All Values' and it is a read only property and cannot be changed after the transformation is created.

Unlock the JOINER Transformation Limitations

Integration Service Configuration

While Informatica PowerCenter Installation

During  PowerCenter installation we can set the supported character set or Data movement Mode  as shown in below image. Please Check out Informatica PowerCenter Installation Guide for step by step installation instruction.

After Informatica PowerCenter Installation

Choose 'Unicode' from the drop down list, and OK.

Read Informatica PowerCenter Installation Guide for the complete Informatica Installation Guide.

Repository Service Configuration

With this configuration, Informatica PowerCenter will have the capability to handle any character with in the UNICODE character set.

I. Analyze the source files

1. Launch Mapping Designer and connect to the repository.
2. Highlight your folder and open the Source Analyzer.
3. To import the ORDERS.TXT flat file definition into the repository, select SOURCES | IMPORT FROM FILE from the menu.
4. Select ORDERS.TXT from the source directory.
   HINT: Be sure to set the Files of type: to All files (*.*) from the pull-down list, before clicking on OK.
5. Set the following options in the Flat File Wizard
   1. Select the Delimited radio button.
   2. Name the source definition ORDERS. Click on Next button.
   3. Enter the column names and specify data types and field widths as shown below.
   4. Click on Finish.
6. Import the PRODUCTS.TXT flat file definition into the repository and name the source definition as PRODUCTS.
7. Verify the column names and change data types as needed to match the preceding table. The source definitions should look like the ones shown below.

II. Design the Target Schema

1. Create the target table definition Tgt_ProductRevenue_x in the Target Designer and generate the SQL script for the same. Your target table definition should look like below image.

III. Drag Sources and Targets into the new Mapping

1. Create a new mapping with the name M_Product_Revenue_x.
2. Set the Mapping Designer options.
3. Automatically create Source Qualifiers when calling source definitions into the mapping.
4. Click OK.
5. Drag the ORDERS and PRODUCTS source definitions from the Navigator Window into the Mapping Designer workspace. Two Source Qualifiers will be created automatically .
6. Accept the default names.
7. Drag the Tgt_ProductRevenue_x target definition from the Navigator Window into the workspace.

IV. Create the Joiner Transformation

1. Choose TRANSFORMATION | CREATE.
2. Select the Joiner transformation from the pull down list.
3. Name it Jnr_Orders_Products_x Make sure you are in link mode by selecting LAYOUT | LINK COLUMNS.
4. Link the ITEM_NO, ITEM_NAME, and PRODUCT_CATEGORY ports from SQ_Products (Source Qualifier) into Jnr_Orders_Products_x (Joiner).
5. Link the ITEM_NO, QTY, and PRICE ports: from SQ_ORDERS (Source Qualifier) into Jnr_Orders_Products_x (Joiner).
6. Open the Joiner transformation in Edit mode.
7. Select the Ports tab.
8. Identify all the ports from PRODUCTS as Master ports.
   HINT: Check the M column checkbox for any one of the ports, which flow originally from the PRODUCTS source definition. 
9. Select the Condition tab.
10. Add a new condition: ITEM_NO = ITEM_NO1.
11. Exit the Edit Transformations dialog box by clicking the OK button.

V. Create the Aggregator Transformation

1. Create an aggregator transformation and name it Agg_ProductRevenue_x.
2. Link the ITEM_NO, ITEM_NAME, PRODUCT_CA TEGORY , QTY , and PRICE from JNR_ORDERS_PRODUCTS (Joiner) into Agg_Product_Revenue_x (Aggregator).
3. Switch to copy mode, LAYOUT | COPY COLUMNS.
4. Copy the TOTAL_QTY and TOTAL_REVENUE ports from Tgt_ProductRevenue_x (target definition) into the Agg_ProductRevenue_x(Aggregator).
   Remainder: You are not linking these ports. Instead you are using the column names (ports) in the target table definition as a model for the names of the ports in the new Aggregator.
5. Group by ITEM_NO, ITEM_NAME, and PRODUCT_CA TEGORY . Hint : Check the Group by columns check box under the Ports tab in the Edit Transformations box.
6. Enter aggregate expressions for the TOTAL_QUANTITY and TOTAL_REVENUE ports and make QTY as input port: TOTAL_QUANTITY: SUM(QTY) TOTAL_REVENUE : SUM(PRICE * QTY)
7. The final transformation should look like the one shown below.

VI. Create the Rank Transformation

1. Choose TRANSFORMA TION | CREA TE.
2. Select Rank from the pull-down list.
3. Name it Rnk_TopTen_x.

   Note: You can also click on the icon from the transformations toolbar to create the rank transformation.
4. Switch to link mode LAYOUT | LINK COLUMNS.
5. Link ITEM_NO, ITEM_NAME, PRODUCT_CATEGORY, PRICE, TOTAL_QUANTITY and TOTAL_REVENUE from Agg_ProductRevenue (Aggregator) into Rnk_TopTen (Rank).
6. Double-click on the Rank Transformation to enter Edit mode.
7. Select the Ports tab.
8. Identify the TOTAL_REVENUE port as the one to rank. Hint: Check the R column. 
9. Deselect the GroupBy options on the ITEM_NAME and PRODUCT_CATEGORY ports.
10. Select the Properties tab.
11. Select Top/Bottom = Top, and Number of Ranks = 10.
12. Exit the Edit Transformations dialog box by clicking the OK button.
13. Connect the Rank transformation to the target table. Hint: Select LAYOUT | AUTOLINK, OR Right click on the workspace and choose Autolink.
14. Select REPOSITORY | SAVE.
15. Given below is the final mapping.

VII. Load the Target

1. Create a Workflow with the name wf_ProductRevenue_x.
   1. Create a session task with the name s_ProductRevenue_x
2. Run the Workflow.
3. Monitor the Workflow.
4. Verify the results for target table Tgt_ProductRevenue_x.

VIII. Video Tutorial

1. Search Tool

Find in Workspace dialog box opens up as sown below. Like shown in the image, you can search for the Fiels, Table (Transformation).  It shows all the instances of the searched column.

2. Auto Linking Ports

You can automatically link ports between transformations either by position or by name. You can invoke this tool from the menu Layout > Autolink or by right clicking the workspace and choosing Autolink.

When you link by position, the Designer links the first output port to the first input port, the second output port to the second input port, and so forth.

You can link ports by name in the Designer. The Designer adds links between input and output ports that have the same name. As shown in below image ports between SRT_GROUP_CUSTOMERS and EXP_FILE_NAME are linked automatically by name.

When you link ports automatically by name, you can specify a prefix or suffix by which to link the ports. Use prefixes or suffixes to indicate where ports occur in a mapping.

For example, a mapping includes a port in the Source Qualifier called CUST_ID and a corresponding port in a Union transformation called CUST_ID1. ‘1’ is the suffix you specify when you automatically link ports between the Source Qualifier and Union transformation.

Like shown in below image you can setup properties in the autolink dialogbox.

Ports from Source Qualifies has been mapped to the Union Transformation as shown in below image.

3. Propagating Port Attributes

You propagate any attributes such port name, datatype, precision, scale, and description as throughout the mapping. For example if you change the precision of a port, you will have to apply the same precision to all remaining transformations as well. In such cases you propagate post attributes.

The Designer propagates ports, expressions, and conditions based on the following factors:

• The direction that you propagate : You can propagate changes forward, backward, or in both directions.
• The attributes you choose to propagate : You can propagate port name, datatype, precision, scale, and description.
• The type of dependencies : You can propagate changes to dependencies along a link path or to implicit dependencies within a transformation.

You can choose the properties mentioned above from the dialogbox shown below. You can invoke this dialogbox by right clicking any transformation ports of a mapping open in the workspace.

You can see from below image, you can see the preview of the port propagation highlighted in green.

4. Viewing Dependencies

Source Column Dependencies : When editing a mapping, you can view source column dependencies for a target column. Viewing source column dependencies lets you see from which source columns a target column receives data.

To view column dependencies, right-click a target column in a mapping and choose Show Field Dependencies.

Here you can see CUST_NAME is dependent on both LAST_NAME, FIRST_NAME column from the Customer source table.

Object Dependencies : You may need to find out the dependent objects for any chosen object. Right click on any object to invoke the Dependency dialogbox shown below.

5. Link Path

When editing a mapping, you may want to view the forward and backward link paths to a particular port. Link paths allow you to see the flow of data from a column in a source, through ports in transformations, to a port in the target.

To view link paths, highlight a port and right click on it. Select the Select Link Path option. You can choose to view either the forward path, backward path, or both as shown in below image.

6. Overview Window

An optional window to simplify viewing workbooks containing large mappings or a large number of objects. Outlines the visible area in the workspace and highlights selected objects in color. This option is really helpful when working with mapping having large number of transformations.

To open the Overview window, click View > Overview Window.

7. Iconizing Workspace Objects

Click Layout > Arrange All Iconic to Iconic the objects.

8. Table Definition Options

Table definition option can be used to specify the properties you see in each object such as transformations, source, target definition. You might need to see the Xpath property of an XML transformation or Level Property of a Normalizer transformation.

You can change these setting using target definition options. Click Tools > Options for this option.

9. Copying Designer Objects

You can copy and paste objects in Informatica PowerCenter Designer tool. These operations can be done across folders as long as both the folders are open. You can use Ctrl + C and Ctrl + V to do these operations or use the Edit Menu.

10. Comparing Objects

You can compare two repository objects of the same type to identify differences between the objects through the Designer. For example, you may want to use a target definition in a mapping, but you have two target definitions that are similar. You can compare the target definitions to see which one contains the columns you need. When you compare two objects, the Designer displays their attributes side-by-side.

What is Regular Expression or RegEx

Regular Expression Example

Regular Expression Implementation in Informatica PowerCenter

What is Checksum

How to find Checksum

Syntax : MD5( value )

Return : Unique 32-character string of hexadecimal digits 0-9 and a-f.

Informatica Implementation

Design Scenario

Lets consider a workflow to load CUSTOMER table from a  flat file, which is generated from legacy mainframe system. Any new customer information will be inserted and any changed customer information will be updated else rejected to bad file. Note that the source file do not have any indicator to identify the changed record.

Datamodel Needs

Apart from the customer attributes columns, we need to create an additional database table column to store the CHECKSUM number. It is 32 character hexadecimal value. So add the column to the CUSTOMER table and below is the target table definition.

Informatica Mapping

Lets start with the mapping. Create CHECKSUM using MD5() function in the expression as sown in below image.

• MD5(CUST_NAME || ADDRESS1 || ADDRESS2 || CITY || STATE || TO_CHAR(ZIP))

Now create a LookUp transformation to get CUST_ID and CHK_SUM_NB from the target table. Use the LookUp Condition IN_CUST_ID = CUST_ID

Now find out the records for INSERT, UPDATE using the columns from the lookUp Transformation with the expressions below.

INSERT : ISNULL(LKP_CUST_ID)
If the Customer is not existing in the target table, set the record for INSERT.

UPDATE : NOT ISNULL(LKP_CUST_ID) AND CHK_SUM_NB <> LKP_CHK_SUM_NB
If the Customer is existing in the target table and the Checksum of the source record is different from the lookup, set the record for UPDATE.

REJECT : Any other records not satisfying the above conditions will be passed on to the DEFAULT group and ignored.

Now add the Router Transformation with two groups with the expressions explained above.

After the Router Transformation is added the mapping looks like in below image.

Map the columns from INSERT and UPDATE group to both target instances, including CHK_SUM_NB to the target table.

Understand the Staging and Dimension Table.

For our demonstration purpose, lets consider the CUSTOMER Dimension. Here we will keep previous version of CITY, STATE, ZIP into its corresponding PREV columns. Below are the detailed structure of both staging and dimension table.

Staging Table

In our staging table, we have all the columns required for the dimension table attributes. So no other tables other than Dimension table will be involved in the mapping. Below is the structure of our staging table.

Key Points 

1. Staging table will have only one days data.
2. Data is uniquely identified using CUST_ID.
3. All attribute required by Dimension Table is available in the staging table

Dimension Table

Here is the structure of our Dimension table.

Key Points

• CREATE_DT :- SYSDATE
• UPDATE_DT :- SYSDATE
• INS_UPD :- IIF(ISNULL(CUST_KEY),'INS', IIF(CITY <> PREV_CITY OR STATE <> PREV_STATE OR ZIP <> PREV_ZIP, 'UPD'))   

INSERT Group

Note : Update Strategy is not required, if the records are set for Insert.

UPDATE Group

Workflow and Session Creation

• BackUp the Repository
• View the BackUps
• Restore the Backup

BackUp the Repository

View the BackUps

Restore the Backup