Run Code

This is an advanced usage.  It permits you to reference a table of values for input variables used to crawl a site, and can use it to control the process flow.

You can write some python code to control the flow or input data.



There are some default code templates in FMiner for some common uses

1. counter [template]

This template will create a counter in variables, every time run the action, it will add 1. It can be used as a looping counter.


2. read table [template]

This template will read each row of a data table and put the data into the variables with name as format of table_column when execute the action.


3. clear table [template]

Clear all data in a table.


4. import data to table [template]

Import XLS/CSV file to a table. The file path can be relative path, then FMiner will try to find the file in folder of project file(*.fmpx).


How to Write Code

The codes are written with python language, and there are some special global variables in FMiner to do work.


Global Variables in Code

1. result

By default, result is True, when you set it to False, the "runcode" action will be done as failure, see the counter template code:

if not variables.has_key("counter1"):
    variables["counter1"] = 0
    variables["counter1"] += 1

if variables["counter1"] >= 10:
    result = False

You can see the line result = False will make the action done as failing to break the loop.


2. variables

The global variables that other run code actions and itself can visit repeatedly. It's a python dict value. All of the variables will show in variables table.


3. tables

This is an global object for control all data tables. see read table template:

table = tables['output']
if not variables.has_key('output_index'):
    variables['output_index'] = 0
curIndex = variables['output_index']
if curIndex >= len(table):
    result = False
    row = table[curIndex]
    for k, v in row.iteritems():
        variables['output_' + k] = v
    variables['output_index'] += 1

In this template code, program will use tables read all rows of the table named "output" and put the data into the variables.


3.1 tables functions

  • __getitem__(self, tabName): Get a table object by its name. e.g tables['input'].
  • table_names(self): Return a list of all tables' name.
  • update_ui(self): Your operation on the tables will just done in background and not show in the UI, call this function to update all UI data tables to show the changes.

3.2 table object functions

tables['table_name'] will return a table object for operate a table, its functions:


  • __getitem__(self, rowId): Get a row of the table, return value is a python dict. (rowId start from 0)
  • __len__(self): Get the rows count of the table.
  • clear(self): Remove all data in the table.
  • remove_row(self, rowId): Remove a row.
  • select_rows(self, filter): filter is a dict. e.g {'name': 'dog'}, it will return a list with all rows name='dog'.
  • add_rows(self, rows): Add some rows, rows should be list with some dict. e.g [{'name':'dog'}, {'name':'cat'}].
  • edit_row(self, rowId, row): Edit the row with rowId, row should be a dict. e.g {'name': 'dog'} (rowId start from 0)
  • __iter__(self): Yield for all rows of the table.
  • __getslice__(self, i, j): Yield some of rows in the table.
  • import_csv(filePath): Import a CSV file to current table.
  • import_xls(filePath): Import a XLS file to current table. 

3.3 Some example

Example 1, change all rows of a table


In this example, the script will change every row of the table, remove first 8 characters, if the data in table like this: "emailto:xxx", when run the script, data will remove the "emailto:":

table = tables['emails']              #get table of "emails"
for i in range(len(table)):           #loop every row of the table
    row = table[i]                    #get a row
    row['mail'] = row['mail'][8:]   #change field of "mail", remove first 8 char
    table.edit_row(i, row)           #write this row

tables.update_ui()                       #update UI to show results


Example 2, write one of variables to a table

In this example, we assume that there is a data named "var1" in global variables, this script will write this variable to the first row of table "config" with field named "cfgvar":

table = tables['config']    #get table of "config"
v = variables['var1']       #read "var1" from global variables
row = {'cfgvar': v}         #create a dict to write to table
table.edit_row(0, row)    #row the data to the first row of the table

tables.update_ui()      #update UI to show results.


Example 3, read a row from a table, then write to another table

table1 = tables['table1']    #get table1
table2 = tables['table2']     #get table2
row = table1[0]                  #get first row of table1
row['data2'] = row['data1'] #change field name for table2
table2.edit_row(0, row)     #write the row to table2

tables.update_ui()   #update UI to show results