![\"OML](\"http://www.oracle.com/a/tech/img/otn-batch1/oml-notebook-tool.jpg\")
Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nCopyright (c) 2021 Oracle Corporation
\nOracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nCopyright (c) 2021 Oracle Corporation
\nOracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis example shows you how to:
\nCopyright (c) 2021 Oracle Corporation
\nThe notebook toolbar contains buttons for running code in paragraphs and for setting configuration and display options.
\nFor example, it displays the current status and the number of users connected to the notebook. It also contains a menu item for keyboard shortcuts and options to show or hide the markdown editor and paragraph output. Additional settings are shown here.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-823691855","id":"20210125-170339_711433015","dateCreated":"2021-01-19T22:17:44+0000","dateStarted":"2021-02-05T20:35:34+0000","dateFinished":"2021-02-05T20:35:34+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%md\n\n![\"OML](\"https://www.oracle.com/a/tech/img/otn-batch1/oml-notebook-bindings.png\")
Oracle Machine Learning notebooks contain an internal list of bindings to fetch data from the database or another data source, such as Oracle Cloud Object Storage. For this lab, we set the interpreter binding to connect to the ADW database and run queries.
\nClick the interpreter bindings icon in the upper right-corner of the Noteook to view the available interpreter bindings.
\nThe default service is low. Click to bind or unbind an interpreter. Drag-and-drop individual interpreter binding settings up or down to order which binding will be used by default. The first interpeter on the list becomes the default. Those highlighted in blue are active.
\nFor more information on Autonomous Database service levels, refer to the Autonomous Databse User's Guide.
\nAn interpreter is a plug-in that allows you to use a specific data processing\n
language in your Oracle Machine Learning notebook. You can add multiple paragraphs, and each paragraph can be connected to different interpreters such as SQL or Python.
We create different paragraphs with different interpreters based on the code we want to run in the paragraphs. The interpreter is set at the top of the paragraph.
\nThe available interpreters are:
\nUse the Markdown interpreter:
\nPaste the Markdown code below into the following %md paragraph. Then click on the arrow to run the code. Note that it displays the formatted text, and adds a new paragraph. Notebooks save automatically, there is no need to click a save button.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_1329918445","id":"20210125-170339_1407141092","dateCreated":"2021-01-12T05:25:21+0000","dateStarted":"2021-02-05T20:35:36+0000","dateFinished":"2021-02-05T20:35:36+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:45"},{"text":"#### Oracle Machine Learning for Python Live Lab\nWelcome to the **OML4Py** Live Lab!\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:37+0000","config":{"colWidth":6,"fontSize":9,"enabled":false,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/osql","editorHide":false},"settings":{"params":{},"forms":{}},"interrupted":false,"jobName":"paragraph_1611594219658_-824398990","id":"20210125-170339_1767565102","dateCreated":"2021-01-12T20:47:49+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"commited":false,"$$hashKey":"object:46"},{"text":"%md\n\n\n\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:37+0000","config":{"colWidth":6,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":false},"settings":{"params":{},"forms":{}},"interrupted":false,"jobName":"paragraph_1611594219658_-889946946","id":"20210125-170339_1448334790","dateCreated":"2021-01-12T05:26:52+0000","dateStarted":"2021-01-12T20:50:04+0000","dateFinished":"2021-01-12T20:50:04+0000","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:47"},{"title":"Enable Python interpreter","text":"%md\n\nTo connect to the Python interpreter, we use **%python** at the top of a new paragraph in the notebook.\n\nRun a simple addition command in Python and select the arrow to run the paragraph. Alternatively, you can press the **Shift+Enter** keys to run commands in the notebook.","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:37+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"To connect to the Python interpreter, we use %python at the top of a new paragraph in the notebook.
\nRun a simple addition command in Python and select the arrow to run the paragraph. Alternatively, you can press the Shift+Enter keys to run commands in the notebook.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-1112594032","id":"20210125-170339_768967667","dateCreated":"2021-01-08T18:40:36+0000","dateStarted":"2021-02-05T20:35:37+0000","dateFinished":"2021-02-05T20:35:37+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:48"},{"text":"%python\n\n1+1\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:37+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"2\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-673203002","id":"20210125-170339_75762377","dateCreated":"2021-01-08T19:48:29+0000","dateStarted":"2021-02-05T20:35:38+0000","dateFinished":"2021-02-05T20:35:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:49"},{"title":"Import the oml package","text":"%md\n\nTo use OML4Py, import the **oml** package. This automatically establishes a connection to the Autonomous database.\n\nThe notebook connects to the Python interpreter, and you are ready to run Python and OML4Py commands in your notebook.\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:38+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"To use OML4Py, import the oml package. This automatically establishes a connection to the Autonomous database.
\nThe notebook connects to the Python interpreter, and you are ready to run Python and OML4Py commands in your notebook.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_2125658335","id":"20210125-170339_978976480","dateCreated":"2021-01-08T19:48:40+0000","dateStarted":"2021-02-05T20:35:38+0000","dateFinished":"2021-02-05T20:35:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:50"},{"text":"%python\n\nimport oml","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:39+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/undefined","fontSize":9,"results":{},"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594219658_-1377868160","id":"20210125-170339_1928130952","dateCreated":"2021-01-08T17:49:03+0000","dateStarted":"2021-02-05T20:35:39+0000","dateFinished":"2021-02-05T20:35:39+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:51"},{"title":"Verify Autonomous Database connection","text":"%md\n\n\nRun the ***oml.isconnected()*** command to verify if the oml module is connected to the Autonomous Database. Once your notebook is connected, the command returns *True*. \n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:39+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Run the oml.isconnected() command to verify if the oml module is connected to the Autonomous Database. Once your notebook is connected, the command returns True.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_1865015923","id":"20210125-170339_1931951677","dateCreated":"2021-01-08T18:05:44+0000","dateStarted":"2021-02-05T20:35:40+0000","dateFinished":"2021-02-05T20:35:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:52"},{"text":"%python\n\noml.isconnected()\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"True\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-1061656716","id":"20210125-170339_1473360082","dateCreated":"2021-01-08T17:55:44+0000","dateStarted":"2021-02-05T20:35:40+0000","dateFinished":"2021-02-05T20:35:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"title":"View help files","text":"%md\n\nThe Python ***help*** function is used to display the documentation of packages, modules, functions, classes, and keywords. The help function has the following syntax:\n\n help([object])\n\nFor example, the ***oml.create*** function creates persistent table in your database schema from data in your Python session. To view the help file for ***oml.create***, do the following:","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":false,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"The Python help function is used to display the documentation of packages, modules, functions, classes, and keywords. The help function has the following syntax:
\nhelp([object])\nFor example, the oml.create function creates persistent table in your database schema from data in your Python session. To view the help file for oml.create, do the following:
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_469021300","id":"20210125-170339_571144758","dateCreated":"2021-01-12T23:07:51+0000","dateStarted":"2021-02-05T20:35:41+0000","dateFinished":"2021-02-05T20:35:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%python\n\nhelp(oml.create)\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on cython_function_or_method in module oml.core.methods:\n\ncreate(x, table, oranumber=True, dbtypes=None, append=False)\n Creates a table in Oracle Database from a Python data set.\n \n Parameters\n ----------\n x : pandas.DataFrame or a list of tuples of equal size\n If ``x`` is a list of tuples of equal size, each tuple represents\n a row in the table. The column names are set to COL1, COL2, ... and so on.\n table : str\n A name for the table.\n oranumber : bool, True (default)\n If True, use SQL NUMBER for numeric columns. Otherwise, use BINARY_DOUBLE.\n Ignored if ``append`` is True.\n dbtypes : dict mapping str to str or list of str\n A list of SQL types to use on the new table. If a list, its length should\n be equal to the number of columns. If a dict, the keys are the names of the\n columns. Ignored if ``append`` is True.\n append : bool, False (default)\n Indicates whether to append the data to the existing table.\n \n Notes\n -----\n * When creating a new table, for columns whose SQL types are not specified in\n ``dbtypes``, NUMBER is used for numeric columns when ``oranumber`` is True\n and BINARY_DOUBLE is used when ``oranumber`` is False. Users should set\n ``oranumber`` to False when the data contains NaN values. For string columns,\n the default type is VARCHAR2(4000), and for bytes columns, the default type\n is BLOB.\n * When ``x`` is specified with an empty pandas.DataFrame, OML creates an\n empty table. NUMBER is used for numeric columns when ``oranumber`` is True\n and BINARY_DOUBLE is used when ``oranumber`` is False. VARCHAR2(4000) is\n used for columns of object dtype in the pandas.DataFrame.\n * OML does not support columns containing values of multiple data types,\n data conversion is needed or a TypeError may be raised.\n * OML determines default column types by looking at 20 random rows sampled\n from the table. For tables with less than 20 rows, all rows are used\n in column type determination. NaN values are considered as float type.\n If a column has all Nones, or has inconsistent data types that are not\n None in the sampled rows, a default column type cannot be determined,\n and a ValueError is raised unless a SQL type for the column is specified\n in ``dbtypes``.\n \n Returns\n -------\n new_table : oml.DataFrame\n A proxy object that represents the newly-created table.\n\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-117335030","id":"20210125-170339_939474766","dateCreated":"2021-01-12T23:11:03+0000","dateStarted":"2021-02-05T20:35:41+0000","dateFinished":"2021-02-05T20:35:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:55"},{"title":"View help for the oml package","text":"%python\n\nhelp(oml)","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true,"editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on package oml:\n\nNAME\n oml - Oracle Machine Learning for Python\n\nDESCRIPTION\n A component of the Oracle Advanced Analytics Option, Oracle Machine Learning\n for Python makes the open source Python programming language and environment\n ready for enterprise in-database data. Designed for problems involving both\n large and small volumes of data, Oracle Machine Learning for Python integrates\n Python with Oracle Database. Python users can run Python commands and scripts\n for statistical, machine learning, and graphical analyses on data stored in\n Oracle Database. Python users can develop, refine, and deploy Python scripts\n that leverage the parallelism and scalability of Oracle Database to automate\n data analysis. Data analysts and data scientists can run Python modules and\n develop and operationalize Python scripts for machine learning applications\n in one step without having to learn SQL. Oracle Machine Learning for Python\n performs function pushdown for in-database execution of core Python and\n popular Python module functions. Being integrated with Oracle Database,\n Oracle Machine Learning for Python can run any Python module via embedded\n Python while the database manages the data served to the Python engines.\n\nPACKAGE CONTENTS\n algo (package)\n automl (package)\n core (package)\n ds (package)\n embed (package)\n graphics (package)\n mlx (package)\n script (package)\n\nCLASSES\n oml.algo.model.odmModel(builtins.object)\n oml.algo.ai.ai\n oml.algo.ar.ar\n oml.algo.dt.dt\n oml.algo.em.em\n oml.algo.esa.esa\n oml.algo.glm.glm\n oml.algo.km.km\n oml.algo.nb.nb\n oml.algo.nn.nn\n oml.algo.rf.rf\n oml.algo.svd.svd\n oml.algo.svm.svm\n oml.core.number._Number(oml.core.series._Series)\n oml.core.float.Float\n oml.core.series._Series(oml.core.vector._Vector)\n oml.core.boolean.Boolean\n oml.core.bytes.Bytes\n oml.core.string.String\n oml.core.vector._Vector(builtins.object)\n oml.core.frame.DataFrame\n \n class Boolean(oml.core.series._Series)\n | Boolean series data class.\n | \n | Represents a single column of 0, 1, and NULL values in Oracle Database.\n | \n | Method resolution order:\n | Boolean\n | oml.core.series._Series\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | __and__(self, other)\n | \n | __init__(self)\n | \n | __invert__(self)\n | \n | __or__(self, other)\n | \n | all(self)\n | Checks whether all elements in the Boolean series data object are True.\n | \n | Returns\n | =======\n | all: bool\n | \n | any(self)\n | Checks whether any elements in the Boolean series data object are True.\n | \n | Returns\n | -------\n | any: bool\n | \n | pull(self)\n | Pulls data represented by this object from Oracle Database into an\n | in-memory Python object.\n | \n | Returns\n | -------\n | pulled_obj : list of bool and None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.series._Series:\n | \n | KFold(self, n_splits=3, seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into k consecutive folds \n | for use with k-fold cross validation.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default):\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of pairs of series objects of the same type as caller\n | Each pair within the list is a fold. The first element of the pair is the\n | train set, and the second element is the test set, which consists of all\n | elements not in the train set.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | __eq__(self, other)\n | Equivalent to ``self == other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | equal : oml.Boolean\n | \n | __ge__(self, other)\n | Equivalent to ``self >= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterequal : oml.Boolean\n | \n | __gt__(self, other)\n | Equivalent to ``self > other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterthan : oml.Boolean\n | \n | __le__(self, other)\n | Equivalent to ``self <= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessequal : oml.Boolean\n | \n | __lt__(self, other)\n | Equivalent to ``self < other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessthan : oml.Boolean\n | \n | __ne__(self, other)\n | Equivalent to ``self != other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | notequal : oml.Boolean\n | \n | count(self)\n | Returns the number of elements that are not NULL.\n | \n | Returns\n | -------\n | nobs : int\n | \n | describe(self)\n | Generates descriptive statistics that summarize the central tendency, \n | dispersion, and shape of an OML series data distribution.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.Series`\n | Includes ``count`` (number of non-null entries), ``unique``\n | (number of unique entries), ``top`` (most common value), ``freq``,\n | (frequency of the most common value).\n | \n | drop_duplicates(self)\n | Removes duplicated elements.\n | \n | Returns\n | -------\n | deduplicated : type of caller\n | \n | dropna(self)\n | Removes missing values.\n | \n | Missing values include None and/or nan if applicable.\n | \n | Returns\n | -------\n | dropped : type of caller\n | \n | isnull(self)\n | Detects the missing value None.\n | \n | Returns\n | -------\n | isnull : oml.Boolean\n | Indicates missing value None for each element.\n | \n | max(self, skipna=True)\n | Returns the maximum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | max : Python type corresponding to the column or numpy.nan\n | \n | min(self, skipna=True)\n | Returns the minimum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | min : Python type corresponding to the column or numpy.nan\n | \n | nunique(self, dropna=True)\n | Returns the number of unique values.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : int\n | \n | sort_values(self, ascending=True, na_position='last')\n | Sorts the values in the series data object.\n | \n | Parameters\n | ----------\n | ascending : bool, True (default)\n | If True, sorts in ascending order. Otherwise, sorts in descending order.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NANs and Nones at the beginning; ``last`` places them\n | at the end.\n | \n | Returns\n | -------\n | sorted_obj : type of caller\n | \n | split(self, ratio=(0.7, 0.3), seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into multiple sets.\n | \n | Parameters\n | ----------\n | ratio : a list of float values or (0.7, 0.3) (default)\n | All the numbers must be positive and the sum of them are no more than \n | 1. Each number represents the ratio of split data in one set. \n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | split_data : a list of series objects of the same type as caller\n | Each of which contains the portion of data by the specified ratio.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | ----------------------------------------------------------------------\n | Data and other attributes inherited from oml.core.series._Series:\n | \n | __hash__ = None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.vector._Vector:\n | \n | __del__(self)\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean\n | Must be from the same data source as self.\n | \n | Returns\n | -------\n | subset : same type as self\n | Contains only the rows satisfying the condition in ``key``.\n | \n | __len__(self)\n | Returns number of rows. Equivalent to ``len(self)``.\n | \n | Returns\n | -------\n | rownum : int\n | \n | __repr__(self)\n | \n | append(self, other, all=True)\n | Appends the ``other`` OML data object of the same class to this data object.\n | \n | Parameters\n | ----------\n | other : An OML data object of the same class\n | all : boolean, True (default)\n | Keeps the duplicated elements from the two data objects.\n | \n | Returns\n | -------\n | appended : type of caller\n | A new data object containing the elements from both objects.\n | \n | concat(self, other, auto_name=False)\n | Combines current OML data object with the ``other`` data objects column-wise.\n | \n | Current object and the ``other`` data objects must be combinable, that is,\n | they both represent data from the same underlying database table, view, or query.\n | \n | Parameters\n | ----------\n | other : an OML data object, a list of OML data objects, or a dict mapping str to OML data objects.\n | * OML data object: an OML series data object or an oml.DataFrame\n | * list: a sequence of OML series and DataFrame objects to concat.\n | * dict: a dict mapping str to OML series and DataFrame objects,\n | the column name of concatenated OML series object is replaced with str,\n | column names of concatenated OML DataFrame object is prefixed with str.\n | Need to specify a :py:obj:`python:collections.OrderedDict` if the \n | concatenation order is expected to follow the key insertion order.\n | auto_name : boolean, False (default)\n | Indicates whether to automatically resolve conflict column names.\n | If True, append duplicated column names with suffix ``[column_index]``.\n | \n | Notes\n | -----\n | After concatenation is done, if there is any empty column names in the resulting \n | oml.DataFrame, they will be renamed with ``COL[column_index]``.\n | \n | Raises\n | ------\n | ValueError\n | * If ``other`` is not a single nor a list/dict of OML objects, or if ``other`` is empty.\n | * If objects in ``other`` are not from same data source.\n | * If ``auto_name`` is False and duplicated column names are detected.\n | \n | Returns\n | -------\n | concat_table : oml.DataFrame\n | An oml.DataFrame data object with its original columns followed by the \n | columns of ``other``.\n | \n | create_view(self, view=None, use_colname=False)\n | Creates an Oracle Database view for the data represented by the OML data \n | object.\n | \n | Parameters\n | ----------\n | view : str or None (default)\n | The name of a database view. If ``view`` is None, the created view is managed\n | by OML and the view is automatically dropped when no longer needed.\n | If a ``view`` is specified, then it is up to the user to drop the view.\n | use_colname : bool, False (default)\n | Indicates whether to create the view with the same column names as the\n | DataFrame object. Ignored if ``view`` is specified.\n | \n | Raises\n | ------\n | TypeError\n | * If the object represents data from a temporary table or view, and the view\n | to create is meant to persist past the current session.\n | \n | Returns\n | -------\n | new_view : oml.DataFrame\n | \n | head(self, n=5)\n | Returns the first n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_head : type of caller\n | \n | materialize(self, table=None)\n | Pushes the contents represented by an OML proxy object (a view, a table and so on) \n | into a table in Oracle Database.\n | \n | Parameters\n | ----------\n | table : str or None (default)\n | The name of a table. If ``table`` is None, an OML-managed table is \n | created, that is, OML drops the table when it is no longer used by \n | any OML object or when you invoke ``oml.disconnect(cleanup=True)`` to \n | terminate the database connection. If a table is specified, \n | then it's up to the user to drop the named table.\n | \n | Returns\n | -------\n | new_table : same type as self\n | \n | tail(self, n=5)\n | Returns the last n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_tail : type of caller\n | \n | ----------------------------------------------------------------------\n | Data descriptors inherited from oml.core.vector._Vector:\n | \n | __dict__\n | dictionary for instance variables (if defined)\n | \n | __weakref__\n | list of weak references to the object (if defined)\n | \n | shape\n | The dimensions of the data set.\n | The first element is the number of rows and the second is the number of\n | columns.\n \n class Bytes(oml.core.series._Series)\n | Bytes(other, dbtype)\n | \n | Binary series data class.\n | \n | Represents a single column of RAW or BLOB data in Oracle Database.\n | \n | Method resolution order:\n | Bytes\n | oml.core.series._Series\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | __init__(self, other, dbtype)\n | Convert underlying Oracle Database type.\n | \n | Parameters\n | ----------\n | other : oml.Bytes\n | dbtype : 'raw' or 'blob'\n | \n | len(self)\n | Computes the length of each byte string.\n | \n | Returns\n | -------\n | length : oml.Float\n | \n | pull(self)\n | Pulls data represented by this object from Oracle Database into an\n | in-memory Python object.\n | \n | Returns\n | -------\n | pulled_obj : list of bytes and None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.series._Series:\n | \n | KFold(self, n_splits=3, seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into k consecutive folds \n | for use with k-fold cross validation.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default):\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of pairs of series objects of the same type as caller\n | Each pair within the list is a fold. The first element of the pair is the\n | train set, and the second element is the test set, which consists of all\n | elements not in the train set.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | __eq__(self, other)\n | Equivalent to ``self == other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | equal : oml.Boolean\n | \n | __ge__(self, other)\n | Equivalent to ``self >= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterequal : oml.Boolean\n | \n | __gt__(self, other)\n | Equivalent to ``self > other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterthan : oml.Boolean\n | \n | __le__(self, other)\n | Equivalent to ``self <= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessequal : oml.Boolean\n | \n | __lt__(self, other)\n | Equivalent to ``self < other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessthan : oml.Boolean\n | \n | __ne__(self, other)\n | Equivalent to ``self != other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | notequal : oml.Boolean\n | \n | count(self)\n | Returns the number of elements that are not NULL.\n | \n | Returns\n | -------\n | nobs : int\n | \n | describe(self)\n | Generates descriptive statistics that summarize the central tendency, \n | dispersion, and shape of an OML series data distribution.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.Series`\n | Includes ``count`` (number of non-null entries), ``unique``\n | (number of unique entries), ``top`` (most common value), ``freq``,\n | (frequency of the most common value).\n | \n | drop_duplicates(self)\n | Removes duplicated elements.\n | \n | Returns\n | -------\n | deduplicated : type of caller\n | \n | dropna(self)\n | Removes missing values.\n | \n | Missing values include None and/or nan if applicable.\n | \n | Returns\n | -------\n | dropped : type of caller\n | \n | isnull(self)\n | Detects the missing value None.\n | \n | Returns\n | -------\n | isnull : oml.Boolean\n | Indicates missing value None for each element.\n | \n | max(self, skipna=True)\n | Returns the maximum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | max : Python type corresponding to the column or numpy.nan\n | \n | min(self, skipna=True)\n | Returns the minimum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | min : Python type corresponding to the column or numpy.nan\n | \n | nunique(self, dropna=True)\n | Returns the number of unique values.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : int\n | \n | sort_values(self, ascending=True, na_position='last')\n | Sorts the values in the series data object.\n | \n | Parameters\n | ----------\n | ascending : bool, True (default)\n | If True, sorts in ascending order. Otherwise, sorts in descending order.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NANs and Nones at the beginning; ``last`` places them\n | at the end.\n | \n | Returns\n | -------\n | sorted_obj : type of caller\n | \n | split(self, ratio=(0.7, 0.3), seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into multiple sets.\n | \n | Parameters\n | ----------\n | ratio : a list of float values or (0.7, 0.3) (default)\n | All the numbers must be positive and the sum of them are no more than \n | 1. Each number represents the ratio of split data in one set. \n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | split_data : a list of series objects of the same type as caller\n | Each of which contains the portion of data by the specified ratio.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | ----------------------------------------------------------------------\n | Data and other attributes inherited from oml.core.series._Series:\n | \n | __hash__ = None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.vector._Vector:\n | \n | __del__(self)\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean\n | Must be from the same data source as self.\n | \n | Returns\n | -------\n | subset : same type as self\n | Contains only the rows satisfying the condition in ``key``.\n | \n | __len__(self)\n | Returns number of rows. Equivalent to ``len(self)``.\n | \n | Returns\n | -------\n | rownum : int\n | \n | __repr__(self)\n | \n | append(self, other, all=True)\n | Appends the ``other`` OML data object of the same class to this data object.\n | \n | Parameters\n | ----------\n | other : An OML data object of the same class\n | all : boolean, True (default)\n | Keeps the duplicated elements from the two data objects.\n | \n | Returns\n | -------\n | appended : type of caller\n | A new data object containing the elements from both objects.\n | \n | concat(self, other, auto_name=False)\n | Combines current OML data object with the ``other`` data objects column-wise.\n | \n | Current object and the ``other`` data objects must be combinable, that is,\n | they both represent data from the same underlying database table, view, or query.\n | \n | Parameters\n | ----------\n | other : an OML data object, a list of OML data objects, or a dict mapping str to OML data objects.\n | * OML data object: an OML series data object or an oml.DataFrame\n | * list: a sequence of OML series and DataFrame objects to concat.\n | * dict: a dict mapping str to OML series and DataFrame objects,\n | the column name of concatenated OML series object is replaced with str,\n | column names of concatenated OML DataFrame object is prefixed with str.\n | Need to specify a :py:obj:`python:collections.OrderedDict` if the \n | concatenation order is expected to follow the key insertion order.\n | auto_name : boolean, False (default)\n | Indicates whether to automatically resolve conflict column names.\n | If True, append duplicated column names with suffix ``[column_index]``.\n | \n | Notes\n | -----\n | After concatenation is done, if there is any empty column names in the resulting \n | oml.DataFrame, they will be renamed with ``COL[column_index]``.\n | \n | Raises\n | ------\n | ValueError\n | * If ``other`` is not a single nor a list/dict of OML objects, or if ``other`` is empty.\n | * If objects in ``other`` are not from same data source.\n | * If ``auto_name`` is False and duplicated column names are detected.\n | \n | Returns\n | -------\n | concat_table : oml.DataFrame\n | An oml.DataFrame data object with its original columns followed by the \n | columns of ``other``.\n | \n | create_view(self, view=None, use_colname=False)\n | Creates an Oracle Database view for the data represented by the OML data \n | object.\n | \n | Parameters\n | ----------\n | view : str or None (default)\n | The name of a database view. If ``view`` is None, the created view is managed\n | by OML and the view is automatically dropped when no longer needed.\n | If a ``view`` is specified, then it is up to the user to drop the view.\n | use_colname : bool, False (default)\n | Indicates whether to create the view with the same column names as the\n | DataFrame object. Ignored if ``view`` is specified.\n | \n | Raises\n | ------\n | TypeError\n | * If the object represents data from a temporary table or view, and the view\n | to create is meant to persist past the current session.\n | \n | Returns\n | -------\n | new_view : oml.DataFrame\n | \n | head(self, n=5)\n | Returns the first n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_head : type of caller\n | \n | materialize(self, table=None)\n | Pushes the contents represented by an OML proxy object (a view, a table and so on) \n | into a table in Oracle Database.\n | \n | Parameters\n | ----------\n | table : str or None (default)\n | The name of a table. If ``table`` is None, an OML-managed table is \n | created, that is, OML drops the table when it is no longer used by \n | any OML object or when you invoke ``oml.disconnect(cleanup=True)`` to \n | terminate the database connection. If a table is specified, \n | then it's up to the user to drop the named table.\n | \n | Returns\n | -------\n | new_table : same type as self\n | \n | tail(self, n=5)\n | Returns the last n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_tail : type of caller\n | \n | ----------------------------------------------------------------------\n | Data descriptors inherited from oml.core.vector._Vector:\n | \n | __dict__\n | dictionary for instance variables (if defined)\n | \n | __weakref__\n | list of weak references to the object (if defined)\n | \n | shape\n | The dimensions of the data set.\n | The first element is the number of rows and the second is the number of\n | columns.\n \n class DataFrame(oml.core.vector._Vector)\n | DataFrame(other)\n | \n | Tabular dataframe class.\n | \n | Represents multiple columns of Boolean, Bytes, Float, and/or String data.\n | \n | Method resolution order:\n | DataFrame\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | KFold(self, n_splits=3, seed=12345, strata_cols=None, use_hash=True, hash_cols=None, nvl=None)\n | Splits the oml.DataFrame object randomly into k consecutive folds.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | strata_cols : a list of string values or None (default)\n | Names of the columns used for stratification. If None, stratification\n | is not performed. Must be None when ``use_hash`` is False.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | hash_cols : a list of string values or None (default)\n | If a list of string values, use the values from these named columns\n | to hash to split the data. If None, use the values from the 1st 10\n | columns to hash.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of k 2-tuples of oml.DataFrame objects\n | \n | Raises\n | ------\n | ValueError\n | * If ``hash_cols`` refers to a single LOB column.\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean, str, list of str, 2-tuple\n | * oml.Boolean : select only the rows satisfying the condition. Must be from the same data\n | source as self.\n | * str : select the column of the same name\n | * list of str : select the columns whose names matches the elements in the list.\n | * 2-tuple : The first element in the tuple denotes which rows to select.\n | It can be either a oml.Boolean or ``slice(None)`` (this selects all\n | rows). The second element in the tuple denotes which columns to select.\n | It can be either ``slice(None)`` (this selects all columns), str, list\n | of str, int, or list of int. If int or list of int, selects the\n | column(s) in the corresponding position(s).\n | \n | Returns\n | -------\n | subset : OML data object\n | Is a oml.DataFrame if has more than one column, otherwise is a OML series data object.\n | \n | __init__(self, other)\n | Convert OML series data object(s) to oml.DataFrame.\n | \n | Parameters\n | ----------\n | other : OML series data object or dict mapping str to OML series data objects\n | * OML series data object : initializes a single-column oml.DataFrame containing the\n | same data. \n | * dict : initializes a oml.DataFrame that comprises all the OML series data objects\n | in the dict in an arbitrary order. Each column in the resulting oml.DataFrame has as\n | its column name its corresponding key in the dict.\n | \n | corr(self, method='pearson', min_periods=1, skipna=True)\n | Computes pairwise correlation between all columns where possible,\n | given the type of coefficient.\n | \n | Parameters\n | ----------\n | method : 'pearson' (default), 'kendall', or 'spearman'\n | * pearson : Uses Pearson's correlation coefficient. Can only calculate\n | correlations between Float or Boolean columns.\n | * kendall : Uses Kendall's tau-b coefficient.\n | * spearman : Uses Spearman's rho coefficient.\n | min_periods : int, optional, 1 (default)\n | The minimum number of observations required per pair of columns to \n | have a valid result.\n | skipna : bool, True (default)\n | If True, NaN and (+/-)Inf values are mapped to NULL.\n | \n | Returns\n | -------\n | y : :py:class:`pandas.DataFrame`\n | \n | count(self, numeric_only=False)\n | Returns the number of elements that are not NULL for each column.\n | \n | Parameters\n | ----------\n | numeric_only : boolean, False (default)\n | Includes only Float and Boolean columns.\n | \n | Returns\n | -------\n | count : :py:class:`pandas.Series`\n | \n | crosstab(self, index, columns=None, values=None, rownames=None, colnames=None, aggfunc=None, margins=False, margins_name='All', dropna=True, normalize=False, pivot=False)\n | Computes a simple cross-tabulation of two or more columns. By default,\n | computes a frequency table for the columns unless a column and\n | an aggregation function have been passed.\n | \n | Parameters\n | ----------\n | index : str or list of str\n | Names of the column(s) of the DataFrame to group by. If ``pivot`` is\n | True, these columns are displayed in the rows of the result table.\n | columns : str or list of str, optional\n | Names of the other column(s) of the Dataframe to group by. If ``pivot``\n | is True, these columns are displayed in the columns of the result\n | table.\n | values : str, optional\n | The name of the column to aggregate according to the grouped columns.\n | Requires ``aggfunc`` to be specified.\n | aggfunc : OML DataFrame aggregation function object, optional\n | The supported oml.DataFrame aggregation functions include: count, \n | max, mean, median, min, nunique, std and sum. To use ``aggfunc``, \n | specify the function object using its full name, for example, \n | ``oml.DataFrame.sum``, ``oml.DataFrame.nunique``, and so on.\n | If specified, requires ``values`` to also be specified.\n | rownames : str or list of str, None (default)\n | If specified, must match number of names in ``index``. If None, names in\n | ``index`` are used. \n | colnames : str or list of str, None (default)\n | If specified, must match number of strings in ``columns``. If None,\n | names in ``columns`` are used. Ignored if ``pivot`` is True.\n | margins : bool, False (default)\n | Includes row and column margins (subtotals)\n | margins_name : str, 'All' (default)\n | Names of the row and column that contain the totals when ``margins``\n | is True. Should be a value not contained in any of the columns specified\n | by ``index`` and ``columns``. \n | dropna : bool, True (default)\n | In addition, if ``pivot`` is True, drops columns from the result\n | table if all the entries of the column are NaN.\n | normalize : boolean, {'all', 'index', 'columns'} or {0, 1}, False (default)\n | Normalizes by dividing the values by their sum.\n | \n | * If 'all' or True, normalizes over all values.\n | * If 'index' or 0, normalizes over each row.\n | * If 'columns' or 1, normalizes over each column.\n | * If ``margins`` is True, also normalizes margin values.\n | pivot : bool, False (default)\n | If True, returns results in pivot table format. Else, returns results in\n | relational table format.\n | \n | Returns\n | -------\n | crosstab : oml.DataFrame\n | \n | See Also\n | --------\n | DataFrame.pivot_table\n | \n | cumsum(self, by, ascending=True, na_position='last', skipna=True)\n | Gets the cumulative sum of each ``Float`` or ``Boolean`` column after the\n | ``DataFrame`` object is sorted.\n | \n | Parameters\n | ----------\n | by : str or list of str\n | A single column name or list of column names by which to sort the \n | DataFrame object. Columns in ``by`` do not have to be ``Float`` or \n | ``Boolean``.\n | ascending : bool or list of bool, True (default)\n | If True, sort is in ascending order, otherwise descending. Specify \n | list for multiple sort orders. If this is a list of bools, must match\n | the length of ``by``.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NaN and None at the beginning, ``last`` places them \n | at the end.\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | cumsum : oml.DataFrame\n | \n | describe(self, percentiles=None, include=None, exclude=None)\n | Generates descriptive statistics that summarize the central tendency,\n | dispersion, and shape of the data in each column\n | \n | Parameters\n | ----------\n | percentiles : bool, list-like of numbers, or None (default), optional \n | The percentiles to include in the output for `Float` columns. All\n | must be between 0 and 1. If ``percentiles`` is None or True,\n | ``percentiles`` is set to ``[.25, .5, .75]``, which corresponds\n | to the 25th, 50th, and 75th percentiles. If `percentiles` is False,\n | only ``min`` and ``max`` stats and no other percentiles is\n | included.\n | include : 'all', list-like of OML column types or None (default), optional\n | Types of columns to include in the result. Available options:\n | \n | - 'all': Includes all columns.\n | - List of OML column types : Only includes specified types in\n | the results.\n | - None (default) : If ``Float`` columns exist and ``exclude`` is\n | None, only includes ``Float`` columns. Otherwise, includes all\n | columns.\n | exclude : list of OML column types or None (default), optional\n | Types of columns to exclude from the result. Available options:\n | \n | - List of OML column types : Excludes specified types from\n | the results.\n | - None (default) : Result excludes nothing.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.DataFrame`\n | The concatenation of the summary statistics for each column.\n | \n | See Also\n | --------\n | DataFrame.count\n | DataFrame.max\n | DataFrame.min\n | DataFrame.mean\n | DataFrame.std\n | DataFrame.select_types\n | \n | drop(self, columns)\n | Drops specified columns.\n | \n | Parameters\n | ----------\n | columns : str or list of str\n | Columns to drop from the object.\n | \n | Returns\n | -------\n | dropped : oml.DataFrame\n | \n | drop_duplicates(self, subset=None)\n | Removes duplicated rows from oml.DataFrame object.\n | \n | Use ``subset`` to consider a set of rows duplicates if they have\n | identical values for only a subset of the columns. In this case, after\n | deduplication, each of the other columns contains the minimum value\n | found across the set.\n | \n | Parameters\n | ----------\n | subset : str or list of str, optional\n | Columns to consider for identifying duplicates. If None, use all\n | columns.\n | \n | Returns\n | -------\n | deduplicated : oml.DataFrame\n | \n | dropna(self, how='any', thresh=None, subset=None)\n | Removes rows containing missing values.\n | \n | Parameters\n | ----------\n | how : {'any', 'all'}, 'any' (default)\n | Determines if row is removed from DataFrame when at least one or all\n | values are missing.\n | thresh : int, optional\n | Requires that many of missing values to drop a row from DataFrame.\n | subset : list, optional\n | The names of the columns to check for missing values.\n | \n | Returns\n | -------\n | dropped : oml.DataFrame\n | DataFrame without missing values.\n | \n | kurtosis(self, skipna=True)\n | Returns the sample kurtosis of the values for each ``Float``\n | column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | \n | Returns\n | -------\n | kurt : :py:class:`pandas.Series`\n | \n | max(self, skipna=True, numeric_only=False)\n | Returns the maximum value in each column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | numeric_only : boolean, False (default)\n | Includes only ``Float`` and ``Boolean`` columns. \n | \n | Returns\n | -------\n | max : :py:class:`pandas.Series`\n | \n | mean(self, skipna=True)\n | Returns the mean of the values for each ``Float`` or ``Boolean`` column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | \n | Returns\n | -------\n | mean : :py:class:`pandas.Series`\n | \n | median(self, skipna=True)\n | Returns the median of the values for each ``Float`` column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Exclude NaN values when computing the result\n | \n | Returns\n | -------\n | median : :py:class:`pandas.Series`\n | \n | merge(self, other, on=None, left_on=None, right_on=None, how='left', suffixes=('_l', '_r'), nvl=True)\n | Joins data sets.\n | \n | Parameters\n | ----------\n | other : an OML data set object\n | on : str or list of str, optional\n | Column names to join on. Must be found in both ``self`` and ``other``.\n | left_on : str or list of str, optional\n | Column names of ``self`` to join on.\n | right_on : str or list of str, optional\n | Column names of ``other`` to join on. If specified, must specify the same\n | number of columns as ``left_on``.\n | how : 'left' (default), 'right', 'inner', 'full'\n | * left : left outer join\n | * right : right outer join\n | * full : full outer join\n | * inner : inner join\n | \n | If ``on`` and ``left_on`` are both None, then ``how`` is ignored,\n | and a cross join is performed.\n | suffixes : sequence of length 2\n | Suffix to apply to column names on the left and right side,\n | respectively.\n | nvl : True (default), False, dict \n | * True : join condition includes NULL value\n | * False : join condition excludes NULL value\n | * dict : specifies the values that join columns use in replacement of NULL value with column names as keys\n | \n | Returns\n | -------\n | merged : oml.DataFrame\n | \n | min(self, skipna=True, numeric_only=False)\n | Returns the minimum value in each column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | numeric_only : boolean, False (default)\n | Includes only ``Float`` and ``Boolean`` columns\n | \n | Returns\n | -------\n | min : :py:class:`pandas.Series`\n | \n | nunique(self, dropna=True)\n | Returns number of unique values for each column of DataFrame.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : pandas.Series\n | \n | pivot_table(self, index, columns=None, values=None, aggfunc=View the oml.push help file.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594219658_-250284223","id":"20210125-170339_2112043354","dateCreated":"2021-01-17T04:53:18+0000","dateStarted":"2021-02-05T20:35:42+0000","dateFinished":"2021-02-05T20:35:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:57"},{"text":"%python\n\n# Insert code here\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:35:43+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1612555992026_-1170351957","id":"20210205-201312_344733276","dateCreated":"2021-02-05T20:13:12+0000","dateStarted":"2021-02-05T20:35:43+0000","dateFinished":"2021-02-05T20:35:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:58"},{"title":"Notes","text":"%md\n\n* ***z*** is a reserved zeppelin-context variable and must not be used as a variable in **%python** paragraphs in Oracle Machine Learning Notebooks. In OML notebooks we use the show method ***z.show*** to display Python objects and proxy object content.\n\n* The Oracle Database connectivity package **cx_Oracle** is used to support database access and users can get access to the underlying connection to use **cx_Oracle** directly via the function ***oml.cursor***. Seez is a reserved zeppelin-context variable and must not be used as a variable in %python paragraphs in Oracle Machine Learning Notebooks. In OML notebooks we use the show method z.show to display Python objects and proxy object content.
\nThe Oracle Database connectivity package cx_Oracle is used to support database access and users can get access to the underlying connection to use cx_Oracle directly via the function oml.cursor. See https://cx-oracle.readthedocs.io/en/latest for general cx_Oracle documentation.
\nOracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis example shows you how to:
\nCopyright (c) 2021 Oracle Corporation
\nThe notebook toolbar contains buttons for running code in paragraphs and for setting configuration and display options.
\nFor example, it displays the current status and the number of users connected to the notebook. It also contains a menu item for keyboard shortcuts and options to show or hide the markdown editor and paragraph output. Additional settings are shown here.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_1651145701","id":"20210302-001843_694491385","dateCreated":"2021-01-19T22:17:44+0000","dateStarted":"2021-03-02T18:53:06+0000","dateFinished":"2021-03-02T18:53:06+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%md\n\nOracle Machine Learning notebooks contain an internal list of bindings to fetch data from the database or another data source, such as Oracle Cloud Object Storage. For this lab, we set the interpreter binding to connect to the ADW database and run queries.
\nClick the interpreter bindings icon in the upper right-corner of the Noteook to view the available interpreter bindings.
\nThe default service is low. Click to bind or unbind an interpreter. Drag-and-drop individual interpreter binding settings up or down to order which binding will be used by default. The first interpeter on the list becomes the default. Those highlighted in blue are active.
\nFor more information on Autonomous Database service levels, refer to the Autonomous Databse User's Guide.
\nAn interpreter is a plug-in that allows you to use a specific data processing\n
language in your Oracle Machine Learning notebook. You can add multiple paragraphs, and each paragraph can be connected to different interpreters such as SQL or Python.
We create different paragraphs with different interpreters based on the code we want to run in the paragraphs. The interpreter is set at the top of the paragraph.
\nThe available interpreters are:
\nUse the Markdown interpreter:
\nPaste the Markdown code below into the following %md paragraph. Then click on the arrow to run the code. Note that it displays the formatted text, and adds a new paragraph. Notebooks save automatically, there is no need to click a save button.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_100624001","id":"20210302-001843_272584438","dateCreated":"2021-01-12T05:25:21+0000","dateStarted":"2021-03-02T18:53:08+0000","dateFinished":"2021-03-02T18:53:08+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:45"},{"text":"#### Oracle Machine Learning for Python Live Lab\nWelcome to the **OML4Py** Live Lab!\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:08+0000","config":{"colWidth":6,"fontSize":9,"enabled":false,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/osql","editorHide":false},"settings":{"params":{},"forms":{}},"interrupted":false,"jobName":"paragraph_1614644323263_1856591355","id":"20210302-001843_374899318","dateCreated":"2021-01-12T20:47:49+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"commited":false,"$$hashKey":"object:46"},{"text":"%md\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:08+0000","config":{"colWidth":6,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":false},"settings":{"params":{},"forms":{}},"interrupted":false,"jobName":"paragraph_1614644323263_211768138","id":"20210302-001843_126729608","dateCreated":"2021-01-12T05:26:52+0000","dateStarted":"2021-02-18T21:18:17+0000","dateFinished":"2021-02-18T21:18:17+0000","status":"FINISHED","errorMessage":"","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:47"},{"title":"Enable Python interpreter","text":"%md\n\nTo connect to the Python interpreter, we use **%python** at the top of a new paragraph in the notebook.\n\nRun a simple addition command in Python and select the arrow to run the paragraph. Alternatively, you can press the **Shift+Enter** keys to run commands in the notebook.","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:09+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"To connect to the Python interpreter, we use %python at the top of a new paragraph in the notebook.
\nRun a simple addition command in Python and select the arrow to run the paragraph. Alternatively, you can press the Shift+Enter keys to run commands in the notebook.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_503903484","id":"20210302-001843_2006358107","dateCreated":"2021-01-08T18:40:36+0000","dateStarted":"2021-03-02T18:53:09+0000","dateFinished":"2021-03-02T18:53:09+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:48"},{"text":"%python\n\n1+1\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:09+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"2\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_-1795475466","id":"20210302-001843_498344097","dateCreated":"2021-01-08T19:48:29+0000","dateStarted":"2021-03-02T18:53:10+0000","dateFinished":"2021-03-02T18:53:15+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:49"},{"title":"Import the oml package","text":"%md\n\nTo use OML4Py, import the **oml** package. This automatically establishes a connection to the Autonomous database.\n\nThe notebook connects to the Python interpreter, and you are ready to run Python and OML4Py commands in your notebook.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:15+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"To use OML4Py, import the oml package. This automatically establishes a connection to the Autonomous database.
\nThe notebook connects to the Python interpreter, and you are ready to run Python and OML4Py commands in your notebook.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_460315593","id":"20210302-001843_1336760915","dateCreated":"2021-01-08T19:48:40+0000","dateStarted":"2021-03-02T18:53:16+0000","dateFinished":"2021-03-02T18:53:16+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:50"},{"text":"%python\n\nimport oml","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:16+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/undefined","fontSize":9,"results":{},"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644323263_-1278571336","id":"20210302-001843_623300563","dateCreated":"2021-01-08T17:49:03+0000","dateStarted":"2021-03-02T18:53:16+0000","dateFinished":"2021-03-02T18:53:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:51"},{"title":"Verify Autonomous Database connection","text":"%md\n\nUsing the default interpreter bindings, OML Notebooks automatically establishes a database connection for the notebook. To verify the Python interpreter has established a database connection through the ***oml*** module, run the ***oml.isconnected()*** command. Once your notebook is connected, the command returns *True*. \n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Using the default interpreter bindings, OML Notebooks automatically establishes a database connection for the notebook. To verify the Python interpreter has established a database connection through the oml module, run the oml.isconnected() command. Once your notebook is connected, the command returns True.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_2043630086","id":"20210302-001843_1722929004","dateCreated":"2021-01-08T18:05:44+0000","dateStarted":"2021-03-02T18:53:17+0000","dateFinished":"2021-03-02T18:53:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:52"},{"text":"%python\n\noml.isconnected()\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"True\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_547082654","id":"20210302-001843_1522293299","dateCreated":"2021-01-08T17:55:44+0000","dateStarted":"2021-03-02T18:53:18+0000","dateFinished":"2021-03-02T18:53:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"title":"View help files","text":"%md\n\nThe Python ***help*** function is used to display the documentation of packages, modules, functions, classes, and keywords. The help function has the following syntax:\n\n help([object])\n\nFor example, the ***oml.create*** function creates persistent table in your database schema from data in your Python session. To view the help file for ***oml.create***, do the following:","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"The Python help function is used to display the documentation of packages, modules, functions, classes, and keywords. The help function has the following syntax:
\nhelp([object])\nFor example, the oml.create function creates persistent table in your database schema from data in your Python session. To view the help file for oml.create, do the following:
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_-814184231","id":"20210302-001843_2145844469","dateCreated":"2021-01-12T23:07:51+0000","dateStarted":"2021-03-02T18:53:18+0000","dateFinished":"2021-03-02T18:53:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%python\n\nhelp(oml.create)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on cython_function_or_method in module oml.core.methods:\n\ncreate(x, table, oranumber=True, dbtypes=None, append=False)\n Creates a table in Oracle Database from a Python data set.\n \n Parameters\n ----------\n x : pandas.DataFrame or a list of tuples of equal size\n If ``x`` is a list of tuples of equal size, each tuple represents\n a row in the table. The column names are set to COL1, COL2, ... and so on.\n table : str\n A name for the table.\n oranumber : bool, True (default)\n If True, use SQL NUMBER for numeric columns. Otherwise, use BINARY_DOUBLE.\n Ignored if ``append`` is True.\n dbtypes : dict mapping str to str or list of str\n A list of SQL types to use on the new table. If a list, its length should\n be equal to the number of columns. If a dict, the keys are the names of the\n columns. Ignored if ``append`` is True.\n append : bool, False (default)\n Indicates whether to append the data to the existing table.\n \n Notes\n -----\n * When creating a new table, for columns whose SQL types are not specified in\n ``dbtypes``, NUMBER is used for numeric columns when ``oranumber`` is True\n and BINARY_DOUBLE is used when ``oranumber`` is False. Users should set\n ``oranumber`` to False when the data contains NaN values. For string columns,\n the default type is VARCHAR2(4000), and for bytes columns, the default type\n is BLOB.\n * When ``x`` is specified with an empty pandas.DataFrame, OML creates an\n empty table. NUMBER is used for numeric columns when ``oranumber`` is True\n and BINARY_DOUBLE is used when ``oranumber`` is False. VARCHAR2(4000) is\n used for columns of object dtype in the pandas.DataFrame.\n * OML does not support columns containing values of multiple data types,\n data conversion is needed or a TypeError may be raised.\n * OML determines default column types by looking at 20 random rows sampled\n from the table. For tables with less than 20 rows, all rows are used\n in column type determination. NaN values are considered as float type.\n If a column has all Nones, or has inconsistent data types that are not\n None in the sampled rows, a default column type cannot be determined,\n and a ValueError is raised unless a SQL type for the column is specified\n in ``dbtypes``.\n \n Returns\n -------\n new_table : oml.DataFrame\n A proxy object that represents the newly-created table.\n\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_-2056072090","id":"20210302-001843_1814949005","dateCreated":"2021-01-12T23:11:03+0000","dateStarted":"2021-03-02T18:53:19+0000","dateFinished":"2021-03-02T18:53:19+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:55"},{"title":"View help for the oml package","text":"%python\n\nhelp(oml)","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:19+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true,"editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on package oml:\n\nNAME\n oml - Oracle Machine Learning for Python\n\nDESCRIPTION\n A component of the Oracle Advanced Analytics Option, Oracle Machine Learning\n for Python makes the open source Python programming language and environment\n ready for enterprise in-database data. Designed for problems involving both\n large and small volumes of data, Oracle Machine Learning for Python integrates\n Python with Oracle Database. Python users can run Python commands and scripts\n for statistical, machine learning, and graphical analyses on data stored in\n Oracle Database. Python users can develop, refine, and deploy Python scripts\n that leverage the parallelism and scalability of Oracle Database to automate\n data analysis. Data analysts and data scientists can run Python modules and\n develop and operationalize Python scripts for machine learning applications\n in one step without having to learn SQL. Oracle Machine Learning for Python\n performs function pushdown for in-database execution of core Python and\n popular Python module functions. Being integrated with Oracle Database,\n Oracle Machine Learning for Python can run any Python module via embedded\n Python while the database manages the data served to the Python engines.\n\nPACKAGE CONTENTS\n algo (package)\n automl (package)\n core (package)\n ds (package)\n embed (package)\n graphics (package)\n mlx (package)\n script (package)\n\nCLASSES\n oml.algo.model.odmModel(builtins.object)\n oml.algo.ai.ai\n oml.algo.ar.ar\n oml.algo.dt.dt\n oml.algo.em.em\n oml.algo.esa.esa\n oml.algo.glm.glm\n oml.algo.km.km\n oml.algo.nb.nb\n oml.algo.nn.nn\n oml.algo.rf.rf\n oml.algo.svd.svd\n oml.algo.svm.svm\n oml.core.number._Number(oml.core.series._Series)\n oml.core.float.Float\n oml.core.series._Series(oml.core.vector._Vector)\n oml.core.boolean.Boolean\n oml.core.bytes.Bytes\n oml.core.string.String\n oml.core.vector._Vector(builtins.object)\n oml.core.frame.DataFrame\n \n class Boolean(oml.core.series._Series)\n | Boolean series data class.\n | \n | Represents a single column of 0, 1, and NULL values in Oracle Database.\n | \n | Method resolution order:\n | Boolean\n | oml.core.series._Series\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | __and__(self, other)\n | \n | __init__(self)\n | \n | __invert__(self)\n | \n | __or__(self, other)\n | \n | all(self)\n | Checks whether all elements in the Boolean series data object are True.\n | \n | Returns\n | =======\n | all: bool\n | \n | any(self)\n | Checks whether any elements in the Boolean series data object are True.\n | \n | Returns\n | -------\n | any: bool\n | \n | pull(self)\n | Pulls data represented by this object from Oracle Database into an\n | in-memory Python object.\n | \n | Returns\n | -------\n | pulled_obj : list of bool and None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.series._Series:\n | \n | KFold(self, n_splits=3, seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into k consecutive folds \n | for use with k-fold cross validation.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default):\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of pairs of series objects of the same type as caller\n | Each pair within the list is a fold. The first element of the pair is the\n | train set, and the second element is the test set, which consists of all\n | elements not in the train set.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | __eq__(self, other)\n | Equivalent to ``self == other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | equal : oml.Boolean\n | \n | __ge__(self, other)\n | Equivalent to ``self >= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterequal : oml.Boolean\n | \n | __gt__(self, other)\n | Equivalent to ``self > other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterthan : oml.Boolean\n | \n | __le__(self, other)\n | Equivalent to ``self <= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessequal : oml.Boolean\n | \n | __lt__(self, other)\n | Equivalent to ``self < other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessthan : oml.Boolean\n | \n | __ne__(self, other)\n | Equivalent to ``self != other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | notequal : oml.Boolean\n | \n | count(self)\n | Returns the number of elements that are not NULL.\n | \n | Returns\n | -------\n | nobs : int\n | \n | describe(self)\n | Generates descriptive statistics that summarize the central tendency, \n | dispersion, and shape of an OML series data distribution.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.Series`\n | Includes ``count`` (number of non-null entries), ``unique``\n | (number of unique entries), ``top`` (most common value), ``freq``,\n | (frequency of the most common value).\n | \n | drop_duplicates(self)\n | Removes duplicated elements.\n | \n | Returns\n | -------\n | deduplicated : type of caller\n | \n | dropna(self)\n | Removes missing values.\n | \n | Missing values include None and/or nan if applicable.\n | \n | Returns\n | -------\n | dropped : type of caller\n | \n | isnull(self)\n | Detects the missing value None.\n | \n | Returns\n | -------\n | isnull : oml.Boolean\n | Indicates missing value None for each element.\n | \n | max(self, skipna=True)\n | Returns the maximum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | max : Python type corresponding to the column or numpy.nan\n | \n | min(self, skipna=True)\n | Returns the minimum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | min : Python type corresponding to the column or numpy.nan\n | \n | nunique(self, dropna=True)\n | Returns the number of unique values.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : int\n | \n | sort_values(self, ascending=True, na_position='last')\n | Sorts the values in the series data object.\n | \n | Parameters\n | ----------\n | ascending : bool, True (default)\n | If True, sorts in ascending order. Otherwise, sorts in descending order.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NANs and Nones at the beginning; ``last`` places them\n | at the end.\n | \n | Returns\n | -------\n | sorted_obj : type of caller\n | \n | split(self, ratio=(0.7, 0.3), seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into multiple sets.\n | \n | Parameters\n | ----------\n | ratio : a list of float values or (0.7, 0.3) (default)\n | All the numbers must be positive and the sum of them are no more than \n | 1. Each number represents the ratio of split data in one set. \n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | split_data : a list of series objects of the same type as caller\n | Each of which contains the portion of data by the specified ratio.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | ----------------------------------------------------------------------\n | Data and other attributes inherited from oml.core.series._Series:\n | \n | __hash__ = None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.vector._Vector:\n | \n | __del__(self)\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean\n | Must be from the same data source as self.\n | \n | Returns\n | -------\n | subset : same type as self\n | Contains only the rows satisfying the condition in ``key``.\n | \n | __len__(self)\n | Returns number of rows. Equivalent to ``len(self)``.\n | \n | Returns\n | -------\n | rownum : int\n | \n | __repr__(self)\n | \n | append(self, other, all=True)\n | Appends the ``other`` OML data object of the same class to this data object.\n | \n | Parameters\n | ----------\n | other : An OML data object of the same class\n | all : boolean, True (default)\n | Keeps the duplicated elements from the two data objects.\n | \n | Returns\n | -------\n | appended : type of caller\n | A new data object containing the elements from both objects.\n | \n | concat(self, other, auto_name=False)\n | Combines current OML data object with the ``other`` data objects column-wise.\n | \n | Current object and the ``other`` data objects must be combinable, that is,\n | they both represent data from the same underlying database table, view, or query.\n | \n | Parameters\n | ----------\n | other : an OML data object, a list of OML data objects, or a dict mapping str to OML data objects.\n | * OML data object: an OML series data object or an oml.DataFrame\n | * list: a sequence of OML series and DataFrame objects to concat.\n | * dict: a dict mapping str to OML series and DataFrame objects,\n | the column name of concatenated OML series object is replaced with str,\n | column names of concatenated OML DataFrame object is prefixed with str.\n | Need to specify a :py:obj:`python:collections.OrderedDict` if the \n | concatenation order is expected to follow the key insertion order.\n | auto_name : boolean, False (default)\n | Indicates whether to automatically resolve conflict column names.\n | If True, append duplicated column names with suffix ``[column_index]``.\n | \n | Notes\n | -----\n | After concatenation is done, if there is any empty column names in the resulting \n | oml.DataFrame, they will be renamed with ``COL[column_index]``.\n | \n | Raises\n | ------\n | ValueError\n | * If ``other`` is not a single nor a list/dict of OML objects, or if ``other`` is empty.\n | * If objects in ``other`` are not from same data source.\n | * If ``auto_name`` is False and duplicated column names are detected.\n | \n | Returns\n | -------\n | concat_table : oml.DataFrame\n | An oml.DataFrame data object with its original columns followed by the \n | columns of ``other``.\n | \n | create_view(self, view=None, use_colname=False)\n | Creates an Oracle Database view for the data represented by the OML data \n | object.\n | \n | Parameters\n | ----------\n | view : str or None (default)\n | The name of a database view. If ``view`` is None, the created view is managed\n | by OML and the view is automatically dropped when no longer needed.\n | If a ``view`` is specified, then it is up to the user to drop the view.\n | use_colname : bool, False (default)\n | Indicates whether to create the view with the same column names as the\n | DataFrame object. Ignored if ``view`` is specified.\n | \n | Raises\n | ------\n | TypeError\n | * If the object represents data from a temporary table or view, and the view\n | to create is meant to persist past the current session.\n | \n | Returns\n | -------\n | new_view : oml.DataFrame\n | \n | head(self, n=5)\n | Returns the first n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_head : type of caller\n | \n | materialize(self, table=None)\n | Pushes the contents represented by an OML proxy object (a view, a table and so on) \n | into a table in Oracle Database.\n | \n | Parameters\n | ----------\n | table : str or None (default)\n | The name of a table. If ``table`` is None, an OML-managed table is \n | created, that is, OML drops the table when it is no longer used by \n | any OML object or when you invoke ``oml.disconnect(cleanup=True)`` to \n | terminate the database connection. If a table is specified, \n | then it's up to the user to drop the named table.\n | \n | Returns\n | -------\n | new_table : same type as self\n | \n | tail(self, n=5)\n | Returns the last n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_tail : type of caller\n | \n | ----------------------------------------------------------------------\n | Data descriptors inherited from oml.core.vector._Vector:\n | \n | __dict__\n | dictionary for instance variables (if defined)\n | \n | __weakref__\n | list of weak references to the object (if defined)\n | \n | shape\n | The dimensions of the data set.\n | The first element is the number of rows and the second is the number of\n | columns.\n \n class Bytes(oml.core.series._Series)\n | Bytes(other, dbtype)\n | \n | Binary series data class.\n | \n | Represents a single column of RAW or BLOB data in Oracle Database.\n | \n | Method resolution order:\n | Bytes\n | oml.core.series._Series\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | __init__(self, other, dbtype)\n | Convert underlying Oracle Database type.\n | \n | Parameters\n | ----------\n | other : oml.Bytes\n | dbtype : 'raw' or 'blob'\n | \n | len(self)\n | Computes the length of each byte string.\n | \n | Returns\n | -------\n | length : oml.Float\n | \n | pull(self)\n | Pulls data represented by this object from Oracle Database into an\n | in-memory Python object.\n | \n | Returns\n | -------\n | pulled_obj : list of bytes and None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.series._Series:\n | \n | KFold(self, n_splits=3, seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into k consecutive folds \n | for use with k-fold cross validation.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default):\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of pairs of series objects of the same type as caller\n | Each pair within the list is a fold. The first element of the pair is the\n | train set, and the second element is the test set, which consists of all\n | elements not in the train set.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | __eq__(self, other)\n | Equivalent to ``self == other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | equal : oml.Boolean\n | \n | __ge__(self, other)\n | Equivalent to ``self >= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterequal : oml.Boolean\n | \n | __gt__(self, other)\n | Equivalent to ``self > other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | greaterthan : oml.Boolean\n | \n | __le__(self, other)\n | Equivalent to ``self <= other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessequal : oml.Boolean\n | \n | __lt__(self, other)\n | Equivalent to ``self < other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | lessthan : oml.Boolean\n | \n | __ne__(self, other)\n | Equivalent to ``self != other``.\n | \n | Parameters\n | ----------\n | other : OML series data object of compatible type or corresponding built-in python scalar\n | * scalar : every element in ``self`` will be compared to the scalar.\n | * a OML series : must come from the same data source. Every element in ``self`` will be\n | compared to the corresponding element in ``other``. oml.Float and oml.Boolean series\n | can be compared to each other. oml.String and oml.Bytes series can be compared to each\n | other.\n | \n | Returns\n | -------\n | notequal : oml.Boolean\n | \n | count(self)\n | Returns the number of elements that are not NULL.\n | \n | Returns\n | -------\n | nobs : int\n | \n | describe(self)\n | Generates descriptive statistics that summarize the central tendency, \n | dispersion, and shape of an OML series data distribution.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.Series`\n | Includes ``count`` (number of non-null entries), ``unique``\n | (number of unique entries), ``top`` (most common value), ``freq``,\n | (frequency of the most common value).\n | \n | drop_duplicates(self)\n | Removes duplicated elements.\n | \n | Returns\n | -------\n | deduplicated : type of caller\n | \n | dropna(self)\n | Removes missing values.\n | \n | Missing values include None and/or nan if applicable.\n | \n | Returns\n | -------\n | dropped : type of caller\n | \n | isnull(self)\n | Detects the missing value None.\n | \n | Returns\n | -------\n | isnull : oml.Boolean\n | Indicates missing value None for each element.\n | \n | max(self, skipna=True)\n | Returns the maximum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | max : Python type corresponding to the column or numpy.nan\n | \n | min(self, skipna=True)\n | Returns the minimum value.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | min : Python type corresponding to the column or numpy.nan\n | \n | nunique(self, dropna=True)\n | Returns the number of unique values.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : int\n | \n | sort_values(self, ascending=True, na_position='last')\n | Sorts the values in the series data object.\n | \n | Parameters\n | ----------\n | ascending : bool, True (default)\n | If True, sorts in ascending order. Otherwise, sorts in descending order.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NANs and Nones at the beginning; ``last`` places them\n | at the end.\n | \n | Returns\n | -------\n | sorted_obj : type of caller\n | \n | split(self, ratio=(0.7, 0.3), seed=12345, use_hash=True, nvl=None)\n | Splits the series data object randomly into multiple sets.\n | \n | Parameters\n | ----------\n | ratio : a list of float values or (0.7, 0.3) (default)\n | All the numbers must be positive and the sum of them are no more than \n | 1. Each number represents the ratio of split data in one set. \n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | split_data : a list of series objects of the same type as caller\n | Each of which contains the portion of data by the specified ratio.\n | \n | Raises\n | ------\n | TypeError\n | * If ``use_hash`` is True, and the underlying database column type of\n | ``self`` is a LOB.\n | \n | ----------------------------------------------------------------------\n | Data and other attributes inherited from oml.core.series._Series:\n | \n | __hash__ = None\n | \n | ----------------------------------------------------------------------\n | Methods inherited from oml.core.vector._Vector:\n | \n | __del__(self)\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean\n | Must be from the same data source as self.\n | \n | Returns\n | -------\n | subset : same type as self\n | Contains only the rows satisfying the condition in ``key``.\n | \n | __len__(self)\n | Returns number of rows. Equivalent to ``len(self)``.\n | \n | Returns\n | -------\n | rownum : int\n | \n | __repr__(self)\n | \n | append(self, other, all=True)\n | Appends the ``other`` OML data object of the same class to this data object.\n | \n | Parameters\n | ----------\n | other : An OML data object of the same class\n | all : boolean, True (default)\n | Keeps the duplicated elements from the two data objects.\n | \n | Returns\n | -------\n | appended : type of caller\n | A new data object containing the elements from both objects.\n | \n | concat(self, other, auto_name=False)\n | Combines current OML data object with the ``other`` data objects column-wise.\n | \n | Current object and the ``other`` data objects must be combinable, that is,\n | they both represent data from the same underlying database table, view, or query.\n | \n | Parameters\n | ----------\n | other : an OML data object, a list of OML data objects, or a dict mapping str to OML data objects.\n | * OML data object: an OML series data object or an oml.DataFrame\n | * list: a sequence of OML series and DataFrame objects to concat.\n | * dict: a dict mapping str to OML series and DataFrame objects,\n | the column name of concatenated OML series object is replaced with str,\n | column names of concatenated OML DataFrame object is prefixed with str.\n | Need to specify a :py:obj:`python:collections.OrderedDict` if the \n | concatenation order is expected to follow the key insertion order.\n | auto_name : boolean, False (default)\n | Indicates whether to automatically resolve conflict column names.\n | If True, append duplicated column names with suffix ``[column_index]``.\n | \n | Notes\n | -----\n | After concatenation is done, if there is any empty column names in the resulting \n | oml.DataFrame, they will be renamed with ``COL[column_index]``.\n | \n | Raises\n | ------\n | ValueError\n | * If ``other`` is not a single nor a list/dict of OML objects, or if ``other`` is empty.\n | * If objects in ``other`` are not from same data source.\n | * If ``auto_name`` is False and duplicated column names are detected.\n | \n | Returns\n | -------\n | concat_table : oml.DataFrame\n | An oml.DataFrame data object with its original columns followed by the \n | columns of ``other``.\n | \n | create_view(self, view=None, use_colname=False)\n | Creates an Oracle Database view for the data represented by the OML data \n | object.\n | \n | Parameters\n | ----------\n | view : str or None (default)\n | The name of a database view. If ``view`` is None, the created view is managed\n | by OML and the view is automatically dropped when no longer needed.\n | If a ``view`` is specified, then it is up to the user to drop the view.\n | use_colname : bool, False (default)\n | Indicates whether to create the view with the same column names as the\n | DataFrame object. Ignored if ``view`` is specified.\n | \n | Raises\n | ------\n | TypeError\n | * If the object represents data from a temporary table or view, and the view\n | to create is meant to persist past the current session.\n | \n | Returns\n | -------\n | new_view : oml.DataFrame\n | \n | head(self, n=5)\n | Returns the first n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_head : type of caller\n | \n | materialize(self, table=None)\n | Pushes the contents represented by an OML proxy object (a view, a table and so on) \n | into a table in Oracle Database.\n | \n | Parameters\n | ----------\n | table : str or None (default)\n | The name of a table. If ``table`` is None, an OML-managed table is \n | created, that is, OML drops the table when it is no longer used by \n | any OML object or when you invoke ``oml.disconnect(cleanup=True)`` to \n | terminate the database connection. If a table is specified, \n | then it's up to the user to drop the named table.\n | \n | Returns\n | -------\n | new_table : same type as self\n | \n | tail(self, n=5)\n | Returns the last n elements.\n | \n | Parameters\n | ----------\n | n : int, 5 (default)\n | The number of elements to return.\n | \n | Returns\n | -------\n | obj_tail : type of caller\n | \n | ----------------------------------------------------------------------\n | Data descriptors inherited from oml.core.vector._Vector:\n | \n | __dict__\n | dictionary for instance variables (if defined)\n | \n | __weakref__\n | list of weak references to the object (if defined)\n | \n | shape\n | The dimensions of the data set.\n | The first element is the number of rows and the second is the number of\n | columns.\n \n class DataFrame(oml.core.vector._Vector)\n | DataFrame(other)\n | \n | Tabular dataframe class.\n | \n | Represents multiple columns of Boolean, Bytes, Float, and/or String data.\n | \n | Method resolution order:\n | DataFrame\n | oml.core.vector._Vector\n | builtins.object\n | \n | Methods defined here:\n | \n | KFold(self, n_splits=3, seed=12345, strata_cols=None, use_hash=True, hash_cols=None, nvl=None)\n | Splits the oml.DataFrame object randomly into k consecutive folds.\n | \n | Parameters\n | ----------\n | n_splits : int, 3 (default)\n | The number of folds. Must be greater than or equal to 2.\n | seed : int or 12345 (default)\n | The seed to use for random splitting.\n | strata_cols : a list of string values or None (default)\n | Names of the columns used for stratification. If None, stratification\n | is not performed. Must be None when ``use_hash`` is False.\n | use_hash : boolean, True (default)\n | If True, use hashing to randomly split the data. If False, use a random\n | number to split the data.\n | hash_cols : a list of string values or None (default)\n | If a list of string values, use the values from these named columns\n | to hash to split the data. If None, use the values from the 1st 10\n | columns to hash.\n | nvl : numeric value, str, or None (default)\n | If not None, the specified values are used to hash in place of Null.\n | \n | Returns\n | -------\n | kfold_data : a list of k 2-tuples of oml.DataFrame objects\n | \n | Raises\n | ------\n | ValueError\n | * If ``hash_cols`` refers to a single LOB column.\n | \n | __getitem__(self, key)\n | Index self. Equivalent to ``self[key]``.\n | \n | Parameters\n | ----------\n | key : oml.Boolean, str, list of str, 2-tuple\n | * oml.Boolean : select only the rows satisfying the condition. Must be from the same data\n | source as self.\n | * str : select the column of the same name\n | * list of str : select the columns whose names matches the elements in the list.\n | * 2-tuple : The first element in the tuple denotes which rows to select.\n | It can be either a oml.Boolean or ``slice(None)`` (this selects all\n | rows). The second element in the tuple denotes which columns to select.\n | It can be either ``slice(None)`` (this selects all columns), str, list\n | of str, int, or list of int. If int or list of int, selects the\n | column(s) in the corresponding position(s).\n | \n | Returns\n | -------\n | subset : OML data object\n | Is a oml.DataFrame if has more than one column, otherwise is a OML series data object.\n | \n | __init__(self, other)\n | Convert OML series data object(s) to oml.DataFrame.\n | \n | Parameters\n | ----------\n | other : OML series data object or dict mapping str to OML series data objects\n | * OML series data object : initializes a single-column oml.DataFrame containing the\n | same data. \n | * dict : initializes a oml.DataFrame that comprises all the OML series data objects\n | in the dict in an arbitrary order. Each column in the resulting oml.DataFrame has as\n | its column name its corresponding key in the dict.\n | \n | corr(self, method='pearson', min_periods=1, skipna=True)\n | Computes pairwise correlation between all columns where possible,\n | given the type of coefficient.\n | \n | Parameters\n | ----------\n | method : 'pearson' (default), 'kendall', or 'spearman'\n | * pearson : Uses Pearson's correlation coefficient. Can only calculate\n | correlations between Float or Boolean columns.\n | * kendall : Uses Kendall's tau-b coefficient.\n | * spearman : Uses Spearman's rho coefficient.\n | min_periods : int, optional, 1 (default)\n | The minimum number of observations required per pair of columns to \n | have a valid result.\n | skipna : bool, True (default)\n | If True, NaN and (+/-)Inf values are mapped to NULL.\n | \n | Returns\n | -------\n | y : :py:class:`pandas.DataFrame`\n | \n | count(self, numeric_only=False)\n | Returns the number of elements that are not NULL for each column.\n | \n | Parameters\n | ----------\n | numeric_only : boolean, False (default)\n | Includes only Float and Boolean columns.\n | \n | Returns\n | -------\n | count : :py:class:`pandas.Series`\n | \n | crosstab(self, index, columns=None, values=None, rownames=None, colnames=None, aggfunc=None, margins=False, margins_name='All', dropna=True, normalize=False, pivot=False)\n | Computes a simple cross-tabulation of two or more columns. By default,\n | computes a frequency table for the columns unless a column and\n | an aggregation function have been passed.\n | \n | Parameters\n | ----------\n | index : str or list of str\n | Names of the column(s) of the DataFrame to group by. If ``pivot`` is\n | True, these columns are displayed in the rows of the result table.\n | columns : str or list of str, optional\n | Names of the other column(s) of the Dataframe to group by. If ``pivot``\n | is True, these columns are displayed in the columns of the result\n | table.\n | values : str, optional\n | The name of the column to aggregate according to the grouped columns.\n | Requires ``aggfunc`` to be specified.\n | aggfunc : OML DataFrame aggregation function object, optional\n | The supported oml.DataFrame aggregation functions include: count, \n | max, mean, median, min, nunique, std and sum. To use ``aggfunc``, \n | specify the function object using its full name, for example, \n | ``oml.DataFrame.sum``, ``oml.DataFrame.nunique``, and so on.\n | If specified, requires ``values`` to also be specified.\n | rownames : str or list of str, None (default)\n | If specified, must match number of names in ``index``. If None, names in\n | ``index`` are used. \n | colnames : str or list of str, None (default)\n | If specified, must match number of strings in ``columns``. If None,\n | names in ``columns`` are used. Ignored if ``pivot`` is True.\n | margins : bool, False (default)\n | Includes row and column margins (subtotals)\n | margins_name : str, 'All' (default)\n | Names of the row and column that contain the totals when ``margins``\n | is True. Should be a value not contained in any of the columns specified\n | by ``index`` and ``columns``. \n | dropna : bool, True (default)\n | In addition, if ``pivot`` is True, drops columns from the result\n | table if all the entries of the column are NaN.\n | normalize : boolean, {'all', 'index', 'columns'} or {0, 1}, False (default)\n | Normalizes by dividing the values by their sum.\n | \n | * If 'all' or True, normalizes over all values.\n | * If 'index' or 0, normalizes over each row.\n | * If 'columns' or 1, normalizes over each column.\n | * If ``margins`` is True, also normalizes margin values.\n | pivot : bool, False (default)\n | If True, returns results in pivot table format. Else, returns results in\n | relational table format.\n | \n | Returns\n | -------\n | crosstab : oml.DataFrame\n | \n | See Also\n | --------\n | DataFrame.pivot_table\n | \n | cumsum(self, by, ascending=True, na_position='last', skipna=True)\n | Gets the cumulative sum of each ``Float`` or ``Boolean`` column after the\n | ``DataFrame`` object is sorted.\n | \n | Parameters\n | ----------\n | by : str or list of str\n | A single column name or list of column names by which to sort the \n | DataFrame object. Columns in ``by`` do not have to be ``Float`` or \n | ``Boolean``.\n | ascending : bool or list of bool, True (default)\n | If True, sort is in ascending order, otherwise descending. Specify \n | list for multiple sort orders. If this is a list of bools, must match\n | the length of ``by``.\n | na_position : {'first', 'last'}, 'last' (default)\n | ``first`` places NaN and None at the beginning, ``last`` places them \n | at the end.\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result.\n | \n | Returns\n | -------\n | cumsum : oml.DataFrame\n | \n | describe(self, percentiles=None, include=None, exclude=None)\n | Generates descriptive statistics that summarize the central tendency,\n | dispersion, and shape of the data in each column\n | \n | Parameters\n | ----------\n | percentiles : bool, list-like of numbers, or None (default), optional \n | The percentiles to include in the output for `Float` columns. All\n | must be between 0 and 1. If ``percentiles`` is None or True,\n | ``percentiles`` is set to ``[.25, .5, .75]``, which corresponds\n | to the 25th, 50th, and 75th percentiles. If `percentiles` is False,\n | only ``min`` and ``max`` stats and no other percentiles is\n | included.\n | include : 'all', list-like of OML column types or None (default), optional\n | Types of columns to include in the result. Available options:\n | \n | - 'all': Includes all columns.\n | - List of OML column types : Only includes specified types in\n | the results.\n | - None (default) : If ``Float`` columns exist and ``exclude`` is\n | None, only includes ``Float`` columns. Otherwise, includes all\n | columns.\n | exclude : list of OML column types or None (default), optional\n | Types of columns to exclude from the result. Available options:\n | \n | - List of OML column types : Excludes specified types from\n | the results.\n | - None (default) : Result excludes nothing.\n | \n | Returns\n | -------\n | summary : :py:class:`pandas.DataFrame`\n | The concatenation of the summary statistics for each column.\n | \n | See Also\n | --------\n | DataFrame.count\n | DataFrame.max\n | DataFrame.min\n | DataFrame.mean\n | DataFrame.std\n | DataFrame.select_types\n | \n | drop(self, columns)\n | Drops specified columns.\n | \n | Parameters\n | ----------\n | columns : str or list of str\n | Columns to drop from the object.\n | \n | Returns\n | -------\n | dropped : oml.DataFrame\n | \n | drop_duplicates(self, subset=None)\n | Removes duplicated rows from oml.DataFrame object.\n | \n | Use ``subset`` to consider a set of rows duplicates if they have\n | identical values for only a subset of the columns. In this case, after\n | deduplication, each of the other columns contains the minimum value\n | found across the set.\n | \n | Parameters\n | ----------\n | subset : str or list of str, optional\n | Columns to consider for identifying duplicates. If None, use all\n | columns.\n | \n | Returns\n | -------\n | deduplicated : oml.DataFrame\n | \n | dropna(self, how='any', thresh=None, subset=None)\n | Removes rows containing missing values.\n | \n | Parameters\n | ----------\n | how : {'any', 'all'}, 'any' (default)\n | Determines if row is removed from DataFrame when at least one or all\n | values are missing.\n | thresh : int, optional\n | Requires that many of missing values to drop a row from DataFrame.\n | subset : list, optional\n | The names of the columns to check for missing values.\n | \n | Returns\n | -------\n | dropped : oml.DataFrame\n | DataFrame without missing values.\n | \n | kurtosis(self, skipna=True)\n | Returns the sample kurtosis of the values for each ``Float``\n | column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | \n | Returns\n | -------\n | kurt : :py:class:`pandas.Series`\n | \n | max(self, skipna=True, numeric_only=False)\n | Returns the maximum value in each column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | numeric_only : boolean, False (default)\n | Includes only ``Float`` and ``Boolean`` columns. \n | \n | Returns\n | -------\n | max : :py:class:`pandas.Series`\n | \n | mean(self, skipna=True)\n | Returns the mean of the values for each ``Float`` or ``Boolean`` column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | \n | Returns\n | -------\n | mean : :py:class:`pandas.Series`\n | \n | median(self, skipna=True)\n | Returns the median of the values for each ``Float`` column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Exclude NaN values when computing the result\n | \n | Returns\n | -------\n | median : :py:class:`pandas.Series`\n | \n | merge(self, other, on=None, left_on=None, right_on=None, how='left', suffixes=('_l', '_r'), nvl=True)\n | Joins data sets.\n | \n | Parameters\n | ----------\n | other : an OML data set object\n | on : str or list of str, optional\n | Column names to join on. Must be found in both ``self`` and ``other``.\n | left_on : str or list of str, optional\n | Column names of ``self`` to join on.\n | right_on : str or list of str, optional\n | Column names of ``other`` to join on. If specified, must specify the same\n | number of columns as ``left_on``.\n | how : 'left' (default), 'right', 'inner', 'full'\n | * left : left outer join\n | * right : right outer join\n | * full : full outer join\n | * inner : inner join\n | \n | If ``on`` and ``left_on`` are both None, then ``how`` is ignored,\n | and a cross join is performed.\n | suffixes : sequence of length 2\n | Suffix to apply to column names on the left and right side,\n | respectively.\n | nvl : True (default), False, dict \n | * True : join condition includes NULL value\n | * False : join condition excludes NULL value\n | * dict : specifies the values that join columns use in replacement of NULL value with column names as keys\n | \n | Returns\n | -------\n | merged : oml.DataFrame\n | \n | min(self, skipna=True, numeric_only=False)\n | Returns the minimum value in each column.\n | \n | Parameters\n | ----------\n | skipna : boolean, True (default)\n | Excludes NaN values when computing the result\n | numeric_only : boolean, False (default)\n | Includes only ``Float`` and ``Boolean`` columns\n | \n | Returns\n | -------\n | min : :py:class:`pandas.Series`\n | \n | nunique(self, dropna=True)\n | Returns number of unique values for each column of DataFrame.\n | \n | Parameters\n | ----------\n | dropna : bool, True (default)\n | If True, NULL values are not included in the count. \n | \n | Returns\n | -------\n | nunique : pandas.Series\n | \n | pivot_table(self, index, columns=None, values=None, aggfunc=View the oml.push help file.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644323263_59725278","id":"20210302-001843_1012546491","dateCreated":"2021-01-17T04:53:18+0000","dateStarted":"2021-03-02T18:53:20+0000","dateFinished":"2021-03-02T18:53:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:57"},{"text":"%python\n\n# Insert code here\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:53:21+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644323263_-2056094775","id":"20210302-001843_1955821969","dateCreated":"2021-02-05T20:13:12+0000","dateStarted":"2021-03-02T18:53:21+0000","dateFinished":"2021-03-02T18:53:21+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:58"},{"title":"Notes","text":"%md\n\n* ***z*** is a reserved zeppelin-context variable and must not be used as a variable in **%python** paragraphs in Oracle Machine Learning Notebooks. In OML notebooks we use the show method ***z.show*** to display Python objects and proxy object content.\n\n* The Oracle Database connectivity package **cx_Oracle** is used to support database access and users can get access to the underlying connection to use **cx_Oracle** directly via the function ***oml.cursor***. Seez is a reserved zeppelin-context variable and must not be used as a variable in %python paragraphs in Oracle Machine Learning Notebooks. In OML notebooks we use the show method z.show to display Python objects and proxy object content.
\nThe Oracle Database connectivity package cx_Oracle is used to support database access and users can get access to the underlying connection to use cx_Oracle directly via the function oml.cursor. See https://cx-oracle.readthedocs.io/en/latest for general cx_Oracle documentation.
\nOracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nWith the transparency layer classes, you can convert selected Python objects to Oracle Autonomous Database (ADB) objects and also call a range of familiar Python functions that are overloaded to run the corresponding SQL on tables in the database.
\nThe OML4Py transparency layer supports functions that interact with database data and enables:
\nNote that using the transparency layer, computations and processing occur in-database, leverage Oracle Database as a high performance compute engine. Whenever possible or appropriate, results are also provided as proxy objects. This eliminates data movement from the database to the Python engine, and is especially valuable for large results.
\nIn this lab, we highlight features of the transparency layer involving data selection and manipulation.
\nCopyright (c) 2021 Oracle Corporation
\nTo use OML4Py, you must first import the oml package. We also import the Pandas package to support OML4Py data manipulation and analysis, data exploration and preparation.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-182313614","id":"20210224-211339_578933342","dateCreated":"2021-01-08T19:59:33+0000","dateStarted":"2021-02-24T21:32:54+0000","dateFinished":"2021-02-24T21:32:54+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%python\n\nimport pandas as pd\nimport oml\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:32:54+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219690_1489255961","id":"20210224-211339_826946900","dateCreated":"2021-01-08T20:27:43+0000","dateStarted":"2021-02-24T21:32:55+0000","dateFinished":"2021-02-24T21:33:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"title":"","text":"%md\n\n.\nThe transparency layer includes the ability to create temporary and permanent Oracle Database objects represented by proxy objects in the Python session. \n\nIn this first example, we load the iris data and combine target and predictors into a single DataFrame, which matches the form the data would have as a database table. This Pandas DataFrame can then be loaded into the database using the ***oml.push*** function, which creates a temporary table and returns a proxy object that we assign to *IRIS_TMP*. \n\nSuch temporary tables will be automatically deleted when the database connection is terminated unless saved in a datastore (the topic of 'datastore' is covered in another lab).\n\nIn OML notebooks we use the zeppelin-context ***z.show*** method to display Python objects and proxy object content. Display the first few rows of *IRIS_TMP* using ***z.show*** for displaying DataFrame results in the Zeppelin viewer. \n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:03+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
The transparency layer includes the ability to create temporary and permanent Oracle Database objects represented by proxy objects in the Python session.
In this first example, we load the iris data and combine target and predictors into a single DataFrame, which matches the form the data would have as a database table. This Pandas DataFrame can then be loaded into the database using the oml.push function, which creates a temporary table and returns a proxy object that we assign to IRIS_TMP.
\nSuch temporary tables will be automatically deleted when the database connection is terminated unless saved in a datastore (the topic of 'datastore' is covered in another lab).
\nIn OML notebooks we use the zeppelin-context z.show method to display Python objects and proxy object content. Display the first few rows of IRIS_TMP using z.show for displaying DataFrame results in the Zeppelin viewer.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_1368915590","id":"20210224-211339_1644305005","dateCreated":"2021-01-08T20:34:04+0000","dateStarted":"2021-02-24T21:33:04+0000","dateFinished":"2021-02-24T21:33:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:43"},{"text":"%python\n\nfrom sklearn.datasets import load_iris\nimport pandas as pd\n\niris = load_iris()\n\nx = pd.DataFrame(iris.data, columns = ['SEPAL_LENGTH','SEPAL_WIDTH', 'PETAL_LENGTH','PETAL_WIDTH'])\ny = pd.DataFrame(list(map(lambda x: {0: 'setosa', 1: 'versicolor', 2:'virginica'}[x], iris.target)), columns = ['SPECIES'])\n \niris_df = pd.concat([x, y], axis=1)\nIRIS_TMP = oml.push(iris_df)\n\nz.show(IRIS_TMP.head())\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:04+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":234,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"SEPAL_LENGTH":"string","SEPAL_WIDTH":"string","PETAL_LENGTH":"string","PETAL_WIDTH":"string","SPECIES":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"SEPAL_LENGTH\tSEPAL_WIDTH\tPETAL_LENGTH\tPETAL_WIDTH\tSPECIES\n4.6\t3.6\t1\t0.2\tsetosa\n5.1\t2.5\t3\t1.1\tversicolor\n6.0\t2.2\t4\t1.0\tversicolor\n5.8\t2.6\t4\t1.2\tversicolor\n5.5\t2.3\t4\t1.3\tversicolor\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-1343707639","id":"20210224-211339_1607839706","dateCreated":"2021-01-08T20:34:41+0000","dateStarted":"2021-02-24T21:33:05+0000","dateFinished":"2021-02-24T21:33:05+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:44"},{"text":"%md\n\n.\nList the available Python transparency layer functions.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:06+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the available Python transparency layer functions.
Select the first three records for columns SEPAL_LENGTH and PETAL_LENGTH from the IRIS_TMP table.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_1532937570","id":"20210224-211339_1632194450","dateCreated":"2021-01-08T20:37:37+0000","dateStarted":"2021-02-24T21:33:07+0000","dateFinished":"2021-02-24T21:33:07+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:47"},{"text":"%python\n\nIRIS_projected1 = IRIS_TMP[:, [\"SEPAL_LENGTH\", \"PETAL_LENGTH\"]]\nIRIS_projected1.head(3)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:08+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" SEPAL_LENGTH PETAL_LENGTH\n0 4.6 1\n1 5.1 3\n2 6.0 4\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_8824237","id":"20210224-211339_319850270","dateCreated":"2021-01-08T20:40:41+0000","dateStarted":"2021-02-24T21:33:08+0000","dateFinished":"2021-02-24T21:33:08+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:48"},{"text":"%md\n\n.\nSelect the first four columns in *IRIS_TMP* using an index range. Note that Python uses 0 as a starting element for indexing. The starting element is the first column, SEPAL_LENGTH. The second element is the second column, SEPAL_WIDTH, and so on. ","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:08+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Select the first four columns in IRIS_TMP using an index range. Note that Python uses 0 as a starting element for indexing. The starting element is the first column, SEPAL_LENGTH. The second element is the second column, SEPAL_WIDTH, and so on.
.\n
Select columns by data type, specifying the data type as oml.string, which returns the SPECIES column.
Perform a simple row selection by selecting columns SEPAL_LENGTH and PETAL_LENGTH where PETAL_LENGTH is less than 1.5.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_87123723","id":"20210224-211339_1165164878","dateCreated":"2021-01-08T20:50:27+0000","dateStarted":"2021-02-24T21:33:12+0000","dateFinished":"2021-02-24T21:33:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"text":"%python\n\nIRIS_filtered1 = IRIS_TMP[IRIS_TMP[\"PETAL_LENGTH\"] < 1.5, [\"SEPAL_LENGTH\", \"PETAL_LENGTH\"]]\n\nprint(\"Length: \", len(IRIS_filtered1))\nIRIS_filtered1.head(3)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:12+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Length: 24\n SEPAL_LENGTH PETAL_LENGTH\n0 4.6 1.0\n1 4.3 1.1\n2 5.8 1.2\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-380948092","id":"20210224-211339_1198650584","dateCreated":"2021-01-08T21:03:51+0000","dateStarted":"2021-02-24T21:33:12+0000","dateFinished":"2021-02-24T21:33:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%md\n\n.\nNow try a compound row selection using the **OR** filtering condition, denoted by the **|** symbol. Select all rows in which PETAL_LENGTH is less than 1.5 OR SEPAL_LENGTH is equal to 5.8.\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:13+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Now try a compound row selection using the OR filtering condition, denoted by the | symbol. Select all rows in which PETAL_LENGTH is less than 1.5 OR SEPAL_LENGTH is equal to 5.8.
.\n
Try a compound row selection using AND, denoted by the & symbol. Select all rows in which PETAL_LENGTH is less than 1.5 AND SEPAL_LENGTH is greater than 5.0.
You can join data from oml.DataFrame objects that represent database tables by using the append, concat, and merge methods.
\nThese steps show how to use these methods.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_2130775286","id":"20210224-211339_1046839004","dateCreated":"2021-01-08T21:09:55+0000","dateStarted":"2021-02-24T21:33:16+0000","dateFinished":"2021-02-24T21:33:16+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:59"},{"title":"Use the append function","text":"%md\n\nThese steps show how to create a temporary table from a Pandas DataFrame and use the ***append*** function. The append argument is a boolean that specifies whether to append the x data to an existing table.\n\nRun the following script to create a temporary table *MY_DF* from a Pandas Dataframe and to print the data type for each column. In addition, this script prints the id column values followed by the num column values by using the append function.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:16+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"These steps show how to create a temporary table from a Pandas DataFrame and use the append function. The append argument is a boolean that specifies whether to append the x data to an existing table.
\nRun the following script to create a temporary table MY_DF from a Pandas Dataframe and to print the data type for each column. In addition, this script prints the id column values followed by the num column values by using the append function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-1990571722","id":"20210224-211339_179825491","dateCreated":"2021-01-08T21:13:15+0000","dateStarted":"2021-02-24T21:33:17+0000","dateFinished":"2021-02-24T21:33:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:60"},{"text":"%python\n\nmy_df = pd.DataFrame({\"id\" : [1, 2, 3, 4, 5],\n \"val\" : [\"a\", \"b\", \"c\", \"d\", \"e\"],\n \"ch\" : [\"p\", \"q\", \"r\", \"a\", \"b\"],\n \"num\" : [4, 3, 6.7, 7.2, 5]})\n\nMY_DF = oml.push(my_df)\n\nprint (my_df.dtypes)\n\nnum1 = MY_DF['id']\nnum2 = MY_DF['num']\nnum1.append(num2)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"id int64\nval object\nch object\nnum float64\ndtype: object\n[1, 2, 3, 4, 5, 4, 3, 6.7, 7.2, 5]\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_1316969416","id":"20210224-211339_1138213452","dateCreated":"2021-01-08T23:19:22+0000","dateStarted":"2021-02-24T21:33:17+0000","dateFinished":"2021-02-24T21:33:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:61"},{"text":"%md\n\n.\nUse the ***append*** function to append an oml.Float series object to another, then append an oml.DataFrame object to another.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the append function to append an oml.Float series object to another, then append an oml.DataFrame object to another.
Use the concat method to combine columns from one OML DataFrame proxy object with those of another. The auto_name argument of the concat method controls whether to call automatic name conflict resolution if one or more column names are duplicates in the two data frames. You can also explicitly rename columns by passing in a dictionary that maps strings to objects, as discussed below.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-2106734090","id":"20210224-211339_489240984","dateCreated":"2021-01-08T23:22:40+0000","dateStarted":"2021-02-24T21:33:19+0000","dateFinished":"2021-02-24T21:33:19+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:64"},{"text":"%md\n\n.\nUse the ***concat*** function to create two oml.DataFrame objects and combine the objects by column. The command automatically prints the result. When there is a single result to show, the print command is not needed.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the concat function to create two oml.DataFrame objects and combine the objects by column. The command automatically prints the result. When there is a single result to show, the print command is not needed.
.\n
An oml.Float is a numeric series data class that represents a single column of NUMBER, BINARY_DOUBLE, or BINARY_FLOAT database data types. Let's create an oml.Float object with the rounded exponential of two times the values in the num column of the oml.frame object, then concatenate it with the oml.DataFrame object y using a new column name.
.\n
Concatenate object x with multiple objects using automatic name conflict resolution. In this example, auto_name=True controls whether to call automatic name conflict resolution if one or more column names are duplicates in the two data frames. In this example, the id and num columns are duplicated.
.\n
Now concatenate multiple OML data objects with customized column renaming. We prepend the word New to the duplicate columns and use OrderedDict to preserve the order in which the objects are added.
Use the merge function to peform a cross join and a left outer join from two objects. This example first performs a cross join on the oml.DataFrame objects x and y, which creates the oml.DataFrame object xy.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_2138982396","id":"20210224-211339_384607938","dateCreated":"2021-01-08T23:52:20+0000","dateStarted":"2021-02-24T21:33:26+0000","dateFinished":"2021-02-24T21:33:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:73"},{"text":"%python\n\nx = MY_DF[['id', 'val']]\ny = MY_DF[['num', 'ch']]\n\nxy = x.merge(y)\nxy\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" id_l val_l num_r ch_r\n0 1 a 4.0 p\n1 2 b 4.0 p\n2 3 c 4.0 p\n3 4 d 4.0 p\n4 5 e 4.0 p\n5 1 a 3.0 q\n6 2 b 3.0 q\n7 3 c 3.0 q\n8 4 d 3.0 q\n9 5 e 3.0 q\n10 1 a 6.7 r\n11 2 b 6.7 r\n12 3 c 6.7 r\n13 4 d 6.7 r\n14 5 e 6.7 r\n15 1 a 7.2 a\n16 2 b 7.2 a\n17 3 c 7.2 a\n18 4 d 7.2 a\n19 5 e 7.2 a\n20 1 a 5.0 b\n21 2 b 5.0 b\n22 3 c 5.0 b\n23 4 d 5.0 b\n24 5 e 5.0 b\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_2131663855","id":"20210224-211339_489687368","dateCreated":"2021-01-09T00:07:04+0000","dateStarted":"2021-02-24T21:33:26+0000","dateFinished":"2021-02-24T21:33:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:74"},{"text":"%md\n\n.\nPerform a left outer join on the first four rows of x with the oml.DataFrame object other on the shared column id and applies the suffixes *.l* and *.r* to column names on the left and right side, respectively.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Perform a left outer join on the first four rows of x with the oml.DataFrame object other on the shared column id and applies the suffixes .l and .r to column names on the left and right side, respectively.
Using the merge help file as a guide, perform a merge with a right outer join on the id column on the left side object x and the num column on the right side object y.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-1248642918","id":"20210224-211339_1856672877","dateCreated":"2021-01-09T01:09:03+0000","dateStarted":"2021-02-24T21:33:28+0000","dateFinished":"2021-02-24T21:33:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:77"},{"text":"%python\n\nhelp(oml.DataFrame.merge)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on cython_function_or_method in module oml.core.frame:\n\nmerge(self, other, on=None, left_on=None, right_on=None, how='left', suffixes=('_l', '_r'), nvl=True)\n Joins data sets.\n \n Parameters\n ----------\n other : an OML data set object\n on : str or list of str, optional\n Column names to join on. Must be found in both ``self`` and ``other``.\n left_on : str or list of str, optional\n Column names of ``self`` to join on.\n right_on : str or list of str, optional\n Column names of ``other`` to join on. If specified, must specify the same\n number of columns as ``left_on``.\n how : 'left' (default), 'right', 'inner', 'full'\n * left : left outer join\n * right : right outer join\n * full : full outer join\n * inner : inner join\n \n If ``on`` and ``left_on`` are both None, then ``how`` is ignored,\n and a cross join is performed.\n suffixes : sequence of length 2\n Suffix to apply to column names on the left and right side,\n respectively.\n nvl : True (default), False, dict \n * True : join condition includes NULL value\n * False : join condition excludes NULL value\n * dict : specifies the values that join columns use in replacement of NULL value with column names as keys\n \n Returns\n -------\n merged : oml.DataFrame\n\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-504628815","id":"20210224-211339_1942862521","dateCreated":"2021-01-09T00:14:12+0000","dateStarted":"2021-02-24T21:33:29+0000","dateFinished":"2021-02-24T21:33:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:78"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:29+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219690_-1087528263","id":"20210224-211339_1630301584","dateCreated":"2021-01-10T17:43:32+0000","dateStarted":"2021-02-24T21:33:29+0000","dateFinished":"2021-02-24T21:33:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:79"},{"title":"Drop rows and columns","text":"%md\n\nIn preparing data for analysis, a typical step is to transform data by dropping some values. You can filter out data that are not needed by using the ***drop***, ***drop_duplicates***, and ***dropna*** methods. \n\nUse the ***del*** statement to remove an OML DataFrame proxy object and its associated temporary table. Note that ***del*** does not delete a persistent table. Use the ***oml.drop*** function to delete a persistent database table. To work with the ***drop*** method, first create a demo data table MY_DF2. Create a temporary table MY_DF2 from Pandas dataframe.\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:30+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"In preparing data for analysis, a typical step is to transform data by dropping some values. You can filter out data that are not needed by using the drop, drop_duplicates, and dropna methods.
\nUse the del statement to remove an OML DataFrame proxy object and its associated temporary table. Note that del does not delete a persistent table. Use the oml.drop function to delete a persistent database table. To work with the drop method, first create a demo data table MY_DF2. Create a temporary table MY_DF2 from Pandas dataframe.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_231633923","id":"20210224-211339_1668324527","dateCreated":"2021-01-09T00:48:50+0000","dateStarted":"2021-02-24T21:33:30+0000","dateFinished":"2021-02-24T21:33:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:80"},{"text":"%python\n\nmy_df2 = pd.DataFrame({'numeric': [1, 1.4, -4, -4, 5.432, None, None],\n 'string1': [None, None, 'a', 'a', 'a', 'b', None],\n 'string2': ['x', None, 'z', 'z', 'z', 'x', None]})\n\n\nMY_DF2 = oml.push(my_df2, dbtypes = {'numeric': 'BINARY_DOUBLE', \n 'string1':'CHAR(1)', \n 'string2':'CHAR(1)'})\n\nMY_DF2\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:30+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" numeric string1 string2\n0 1.400 None None\n1 NaN None None\n2 1.000 None x\n3 NaN b x\n4 -4.000 a z\n5 -4.000 a z\n6 5.432 a z\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_164883936","id":"20210224-211339_1505981776","dateCreated":"2021-01-09T19:46:06+0000","dateStarted":"2021-02-24T21:33:31+0000","dateFinished":"2021-02-24T21:33:31+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:81"},{"text":"%md\n\n.\nDrop rows with missing values.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Drop rows with missing values.
.\n
Drop rows with missing numeric values.
.\n
Drop rows where all columns values are missing.
.\n
Use the drop_duplicates function to drop duplicate rows.
.\n
The following command drops the specified column, string2.
This section demonstrates how to use the split and KFold functions using the digits data set after creating an OML DataFrame proxy object for the digits data set.
\nThe KFold method splits the series data object randomly into k consecutive folds, or non-overlapping groups, for use with k-fold cross validation. The split method splits the series data object randomly into multiple sets.
\nThe following tasks are covered in this section
\n.\n
First, use oml.sync to create a DataFrame proxy object in Python that represents the DIGITS table. The DIGITS table resides in the OMLUSER schema. All database users have been granted read access on the table.
.\n
Split the data set into samples of 80% and 20% size.
.\n
Perform stratified sampling on the column, target. Stratified sampling divides the data into different groups based on shared characteristics prior to sampling, ensuring that members from each subgroup is included in the analysis.
.\n
Verify that the stratified sampling generates splits in which all of the different categories of digits (digits 0~9) are present in each split.
.\n
Compute hash on the target column.
.\n
Verify that the different categories of digits (digits 0~9) are present in only one of the splits generated by hashing on the category column.
.\n
Split the data randomly into 4 consecutive folds using the KFold function.
Split the digit data set into four samples of 25% each.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_1605490423","id":"20210224-211339_1694453352","dateCreated":"2021-01-09T20:21:10+0000","dateStarted":"2021-02-24T21:33:51+0000","dateFinished":"2021-02-24T21:33:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:107"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:51+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219690_1116226404","id":"20210224-211339_730491039","dateCreated":"2021-01-10T17:28:15+0000","dateStarted":"2021-02-24T21:33:51+0000","dateFinished":"2021-02-24T21:33:52+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:108"},{"title":"Step 6: Use the crosstab and pivot_table functions","text":"%md\n\nThis step shows how to use the ***crosstab*** method to perform cross-column analysis of an oml.DataFrame object and the ***pivot_table*** method to convert an oml.DataFrame to a spreadsheet-style pivot table.\n\nCross-tabulation is a statistical technique that finds an interdependent relationship between two columns of values. The ***crosstab*** method computes a cross-tabulation of two or more columns. By default, it computes a frequency table for the columns unless a column and an aggregation function have been passed to it.\n\nThe ***pivot_table*** function converts a data set into a pivot table. Due to the database 1000 column limit, pivot tables with more than 1000 columns are automatically truncated to display the categories with the most entries for each column value.\n\n\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:52+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"This step shows how to use the crosstab method to perform cross-column analysis of an oml.DataFrame object and the pivot_table method to convert an oml.DataFrame to a spreadsheet-style pivot table.
\nCross-tabulation is a statistical technique that finds an interdependent relationship between two columns of values. The crosstab method computes a cross-tabulation of two or more columns. By default, it computes a frequency table for the columns unless a column and an aggregation function have been passed to it.
\nThe pivot_table function converts a data set into a pivot table. Due to the database 1000 column limit, pivot tables with more than 1000 columns are automatically truncated to display the categories with the most entries for each column value.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-546422572","id":"20210224-211339_1567547806","dateCreated":"2021-01-09T20:22:18+0000","dateStarted":"2021-02-24T21:33:52+0000","dateFinished":"2021-02-24T21:33:52+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:109"},{"text":"%md\n\n.\nCreate a temporary table *MY_DF4* using a demo data set. This data set reports on the speed and accuracy for a given task characterized by hand used (left or right) and gender (male, female, or not specified).\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:33:52+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create a temporary table MY_DF4 using a demo data set. This data set reports on the speed and accuracy for a given task characterized by hand used (left or right) and gender (male, female, or not specified).
.\n
Use the crosstab function to find the categories that the most entries belonged to. This example shows how to find the count of gender, and right-handed and left-handed persons in descending order in the MY_DF4 dataframe.
.\n
Use the crosstab function to find the ratio of entries with different hand values for each gender and across entries in the MY_DF4 dataframe.
.\n
Use the pivot_table function to find the mean speed across all gender and hand combinations in the MY_DF4 dataframe.
.\n
Use the pivot_table function to find the maximum and minimum speed for every gender and hand combination and across all combinations in the MY_DF4 dataframe.
OML4Py provides functions for rendering graphical displays of data. The oml.boxplot and oml.hist functions compute the statistics necessary to generate box and whisker plots or histograms in-database for scalability and performance. OML4Py uses the matplotlib library to render the output.
\nThis example demonstrates how to use the oml.boxplot and oml.hist functions using the wine data set. The statistics supporting these plots are computed in the database, so transfer of data or client side memory limitations are avoided. As before, we sync the WINE table in the OMLUSER schema to a local OML DataFrame.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-642451635","id":"20210224-211339_1312608961","dateCreated":"2021-01-09T20:44:55+0000","dateStarted":"2021-02-24T21:34:01+0000","dateFinished":"2021-02-24T21:34:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:120"},{"title":"Create a boxplot","text":"%python\n\nimport matplotlib.pyplot as plt\n\nWINE = oml.sync(table = \"WINE\")\n\noml.graphics.boxplot(WINE[:,8:12], showmeans=True, meanline=True, patch_artist=True, labels=WINE.columns[8:12])\nplt.title('Distribution of Wine Attributes')\nplt.show()\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true,"editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"
Use the oml.create function to create a persistent table in your database schema from data in your Python session. The oml.create function creates a table in the database schema and returns an oml.DataFrame object that is a proxy for the table. The proxy oml.DataFrame object has the same name as the table. The resulting table is available from both Python and SQL, even after the session ends - in contrast to the temporary table created when using the oml.push function
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-1887220161","id":"20210224-211339_1140906918","dateCreated":"2021-01-09T20:50:22+0000","dateStarted":"2021-02-24T21:34:10+0000","dateFinished":"2021-02-24T21:34:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:123"},{"text":"%python\n\ntry:\n oml.drop(table=\"IRIS\")\nexcept:\n pass\n\nIRIS = oml.create(iris_df, table=\"IRIS\")\nprint(\"Shape:\",IRIS.shape)\nz.show(IRIS.head(10))\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:10+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"1":{"graph":{"mode":"table","height":300,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"SEPAL_LENGTH":"string","SEPAL_WIDTH":"string","PETAL_LENGTH":"string","PETAL_WIDTH":"string","SPECIES":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Shape: (150, 5)\n"},{"type":"TABLE","data":"SEPAL_LENGTH\tSEPAL_WIDTH\tPETAL_LENGTH\tPETAL_WIDTH\tSPECIES\n4.6\t3.6\t1\t0.2\tsetosa\n5.1\t2.5\t3\t1.1\tversicolor\n6.0\t2.2\t4\t1.0\tversicolor\n5.8\t2.6\t4\t1.2\tversicolor\n5.5\t2.3\t4\t1.3\tversicolor\n5.5\t2.5\t4\t1.3\tversicolor\n6.1\t2.8\t4\t1.3\tversicolor\n5.7\t2.5\t5\t2.0\tvirginica\n6.0\t2.2\t5\t1.5\tvirginica\n6.3\t2.5\t5\t1.9\tvirginica\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_-60112","id":"20210224-211339_1820500797","dateCreated":"2021-01-10T17:58:51+0000","dateStarted":"2021-02-24T21:34:10+0000","dateFinished":"2021-02-24T21:34:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:124"},{"title":"View persistent tables in the user schema","text":"%python\n\nlist(oml.sync(table=\".*\", regex_match=True).keys())","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:11+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"['CUSTOMERS360_TAB', 'INSUR_CUST_LTV', 'RF_COST', 'CUSTOMERS360', 'BREASTCANCER', 'WINE', 'DIGITS', 'IRIS']\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_235065732","id":"20210224-211339_749189270","dateCreated":"2021-01-14T20:38:36+0000","dateStarted":"2021-02-24T21:34:11+0000","dateFinished":"2021-02-24T21:34:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:125"},{"text":"%md\n\n.\nCreate a cursor object for the database connection using ***oml.cursor*** from **cx_Oracle**, then shows the SQL data types of the columns in table *IRIS*.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:12+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create a cursor object for the database connection using oml.cursor from cx_Oracle, then shows the SQL data types of the columns in table IRIS.
.\n
To close the cursor, run the cr.close command.
.\n
Use the transparency layer function shape to view the shape, or number of rows and columns, of the IRIS table.
.\n
Use the transparency layer function head to view the first three rows of the IRIS table.
Use describe to calculate descriptive statistics that summarize the central tendency, dispersion, and shape of the IRIS table in each numeric column.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_1230271714","id":"20210224-211339_200957615","dateCreated":"2021-01-10T18:35:55+0000","dateStarted":"2021-02-24T21:34:18+0000","dateFinished":"2021-02-24T21:34:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:134"},{"text":"%python\n\nIRIS.describe()\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" SEPAL_LENGTH SEPAL_WIDTH PETAL_LENGTH PETAL_WIDTH\ncount 150.000000 150.000000 150.000000 150.000000\nmean 5.843333 3.057333 3.758000 1.199333\nstd 0.828066 0.435866 1.765298 0.762238\nmin 4.300000 2.000000 1.000000 0.100000\n25% 5.100000 2.800000 1.600000 0.300000\n50% 5.800000 3.000000 4.350000 1.300000\n75% 6.400000 3.300000 5.100000 1.800000\nmax 7.900000 4.400000 6.900000 2.500000\n"}]},"interrupted":false,"jobName":"paragraph_1614201219690_2012181803","id":"20210224-211339_1341337886","dateCreated":"2021-01-10T18:39:20+0000","dateStarted":"2021-02-24T21:34:18+0000","dateFinished":"2021-02-24T21:34:19+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:135"},{"text":"%md\n\n## End of Script\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:34:19+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nWith the transparency layer classes, you can convert selected Python objects to Oracle Autonomous Database (ADB) objects and also call a range of familiar Python functions that are overloaded to run the corresponding SQL on tables in the database.
\nThe OML4Py transparency layer supports functions that interact with database data and enables:
\nNote that using the transparency layer, computations and processing occur in-database, leverage Oracle Database as a high performance compute engine. Whenever possible or appropriate, results are also provided as proxy objects. This eliminates data movement from the database to the Python engine, and is especially valuable for large results.
\nIn this lab, we highlight features of the transparency layer involving data selection and manipulation.
\nCopyright (c) 2021 Oracle Corporation
\nTo use OML4Py, you must first import the oml package. We also import the Pandas package to support OML4Py data manipulation and analysis, data exploration and preparation.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_944456566","id":"20210302-002154_2117648379","dateCreated":"2021-01-08T19:59:33+0000","dateStarted":"2021-03-02T18:55:06+0000","dateFinished":"2021-03-02T18:55:06+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%python\n\nimport pandas as pd\nimport oml\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:06+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644514465_-736551597","id":"20210302-002154_293400369","dateCreated":"2021-01-08T20:27:43+0000","dateStarted":"2021-03-02T18:55:07+0000","dateFinished":"2021-03-02T18:55:15+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"title":"","text":"%md\n\n.\nThe transparency layer includes the ability to create temporary and permanent Oracle Database objects represented by proxy objects in the Python session. \n\nIn this first example, we load the iris data and combine target and predictors into a single DataFrame, which matches the form the data would have as a database table. This Pandas DataFrame can then be loaded into the database using the ***oml.push*** function, which creates a temporary table and returns a proxy object that we assign to *IRIS_TMP*. \n\nSuch temporary tables will be automatically deleted when the database connection is terminated unless saved in a datastore (the topic of 'datastore' is covered in another lab).\n\nIn OML notebooks we use the zeppelin-context ***z.show*** method to display Python objects and proxy object content. Display the first few rows of *IRIS_TMP* using ***z.show*** for displaying DataFrame results in the Zeppelin viewer. \n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:15+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
The transparency layer includes the ability to create temporary and permanent Oracle Database objects represented by proxy objects in the Python session.
In this first example, we load the iris data and combine target and predictors into a single DataFrame, which matches the form the data would have as a database table. This Pandas DataFrame can then be loaded into the database using the oml.push function, which creates a temporary table and returns a proxy object that we assign to IRIS_TMP.
\nSuch temporary tables will be automatically deleted when the database connection is terminated unless saved in a datastore (the topic of 'datastore' is covered in another lab).
\nIn OML notebooks we use the zeppelin-context z.show method to display Python objects and proxy object content. Display the first few rows of IRIS_TMP using z.show for displaying DataFrame results in the Zeppelin viewer.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1138324062","id":"20210302-002154_1660803415","dateCreated":"2021-01-08T20:34:04+0000","dateStarted":"2021-03-02T18:55:16+0000","dateFinished":"2021-03-02T18:55:16+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:43"},{"text":"%python\n\nfrom sklearn.datasets import load_iris\nimport pandas as pd\n\niris = load_iris()\n\nx = pd.DataFrame(iris.data, columns = ['SEPAL_LENGTH','SEPAL_WIDTH', 'PETAL_LENGTH','PETAL_WIDTH'])\ny = pd.DataFrame(list(map(lambda x: {0: 'setosa', 1: 'versicolor', 2:'virginica'}[x], iris.target)), columns = ['SPECIES'])\n \niris_df = pd.concat([x, y], axis=1)\nIRIS_TMP = oml.push(iris_df)\n\nz.show(IRIS_TMP.head())\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:16+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":234,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"SEPAL_LENGTH":"string","SEPAL_WIDTH":"string","PETAL_LENGTH":"string","PETAL_WIDTH":"string","SPECIES":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"SEPAL_LENGTH\tSEPAL_WIDTH\tPETAL_LENGTH\tPETAL_WIDTH\tSPECIES\n4.6\t3.6\t1\t0.2\tsetosa\n5.1\t2.5\t3\t1.1\tversicolor\n6.0\t2.2\t4\t1.0\tversicolor\n5.8\t2.6\t4\t1.2\tversicolor\n5.5\t2.3\t4\t1.3\tversicolor\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1445158550","id":"20210302-002154_124213212","dateCreated":"2021-01-08T20:34:41+0000","dateStarted":"2021-03-02T18:55:16+0000","dateFinished":"2021-03-02T18:55:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:44"},{"text":"%md\n\n.\nList the available Python transparency layer functions.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the available Python transparency layer functions.
Select the first three records for columns SEPAL_LENGTH and PETAL_LENGTH from the IRIS_TMP table.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_178865794","id":"20210302-002154_1188285714","dateCreated":"2021-01-08T20:37:37+0000","dateStarted":"2021-03-02T18:55:19+0000","dateFinished":"2021-03-02T18:55:19+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:47"},{"text":"%python\n\nIRIS_projected1 = IRIS_TMP[:, [\"SEPAL_LENGTH\", \"PETAL_LENGTH\"]]\nIRIS_projected1.head(3)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:19+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" SEPAL_LENGTH PETAL_LENGTH\n0 4.6 1\n1 5.1 3\n2 6.0 4\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_1548826455","id":"20210302-002154_2075638691","dateCreated":"2021-01-08T20:40:41+0000","dateStarted":"2021-03-02T18:55:20+0000","dateFinished":"2021-03-02T18:55:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:48"},{"text":"%md\n\n.\nSelect the first four columns in *IRIS_TMP* using an index range. Note that Python uses 0 as a starting element for indexing. The starting element is the first column, SEPAL_LENGTH. The second element is the second column, SEPAL_WIDTH, and so on. ","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Select the first four columns in IRIS_TMP using an index range. Note that Python uses 0 as a starting element for indexing. The starting element is the first column, SEPAL_LENGTH. The second element is the second column, SEPAL_WIDTH, and so on.
.\n
Select columns by data type, specifying the data type as oml.string, which returns the SPECIES column.
Perform a simple row selection by selecting columns SEPAL_LENGTH and PETAL_LENGTH where PETAL_LENGTH is less than 1.5.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1590696236","id":"20210302-002154_10336898","dateCreated":"2021-01-08T20:50:27+0000","dateStarted":"2021-03-02T18:55:23+0000","dateFinished":"2021-03-02T18:55:23+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"text":"%python\n\nIRIS_filtered1 = IRIS_TMP[IRIS_TMP[\"PETAL_LENGTH\"] < 1.5, [\"SEPAL_LENGTH\", \"PETAL_LENGTH\"]]\n\nprint(\"Length: \", len(IRIS_filtered1))\nIRIS_filtered1.head(3)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:23+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Length: 24\n SEPAL_LENGTH PETAL_LENGTH\n0 4.6 1.0\n1 4.3 1.1\n2 5.8 1.2\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-2087764483","id":"20210302-002154_722451709","dateCreated":"2021-01-08T21:03:51+0000","dateStarted":"2021-03-02T18:55:24+0000","dateFinished":"2021-03-02T18:55:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%md\n\n.\nNow try a compound row selection using the **OR** filtering condition, denoted by the **|** symbol. Select all rows in which PETAL_LENGTH is less than 1.5 OR SEPAL_LENGTH is equal to 5.8.\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:24+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Now try a compound row selection using the OR filtering condition, denoted by the | symbol. Select all rows in which PETAL_LENGTH is less than 1.5 OR SEPAL_LENGTH is equal to 5.8.
.\n
Try a compound row selection using AND, denoted by the & symbol. Select all rows in which PETAL_LENGTH is less than 1.5 AND SEPAL_LENGTH is greater than 5.0.
You can join data from oml.DataFrame objects that represent database tables by using the append, concat, and merge methods.
\nThese steps show how to use these methods.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1893422320","id":"20210302-002154_403552698","dateCreated":"2021-01-08T21:09:55+0000","dateStarted":"2021-03-02T18:55:28+0000","dateFinished":"2021-03-02T18:55:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:59"},{"title":"Use the append function","text":"%md\n\nThese steps show how to create a temporary table from a Pandas DataFrame and use the ***append*** function. The append argument is a boolean that specifies whether to append the x data to an existing table.\n\nRun the following script to create a temporary table *MY_DF* from a Pandas Dataframe and to print the data type for each column. In addition, this script prints the id column values followed by the num column values by using the append function.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"These steps show how to create a temporary table from a Pandas DataFrame and use the append function. The append argument is a boolean that specifies whether to append the x data to an existing table.
\nRun the following script to create a temporary table MY_DF from a Pandas Dataframe and to print the data type for each column. In addition, this script prints the id column values followed by the num column values by using the append function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-706724002","id":"20210302-002154_989220891","dateCreated":"2021-01-08T21:13:15+0000","dateStarted":"2021-03-02T18:55:28+0000","dateFinished":"2021-03-02T18:55:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:60"},{"text":"%python\n\nmy_df = pd.DataFrame({\"id\" : [1, 2, 3, 4, 5],\n \"val\" : [\"a\", \"b\", \"c\", \"d\", \"e\"],\n \"ch\" : [\"p\", \"q\", \"r\", \"a\", \"b\"],\n \"num\" : [4, 3, 6.7, 7.2, 5]})\n\nMY_DF = oml.push(my_df)\n\nprint (my_df.dtypes)\n\nnum1 = MY_DF['id']\nnum2 = MY_DF['num']\nnum1.append(num2)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"id int64\nval object\nch object\nnum float64\ndtype: object\n[1, 2, 3, 4, 5, 4, 3, 6.7, 7.2, 5]\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_1519582561","id":"20210302-002154_1383147791","dateCreated":"2021-01-08T23:19:22+0000","dateStarted":"2021-03-02T18:55:29+0000","dateFinished":"2021-03-02T18:55:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:61"},{"text":"%md\n\n.\nUse the ***append*** function to append an oml.Float series object to another, then append an oml.DataFrame object to another.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:29+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the append function to append an oml.Float series object to another, then append an oml.DataFrame object to another.
Use the concat method to combine columns from one OML DataFrame proxy object with those of another. The auto_name argument of the concat method controls whether to call automatic name conflict resolution if one or more column names are duplicates in the two data frames. You can also explicitly rename columns by passing in a dictionary that maps strings to objects, as discussed below.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1434723039","id":"20210302-002154_1680044376","dateCreated":"2021-01-08T23:22:40+0000","dateStarted":"2021-03-02T18:55:31+0000","dateFinished":"2021-03-02T18:55:31+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:64"},{"text":"%md\n\n.\nUse the ***concat*** function to create two oml.DataFrame objects and combine the objects by column. The command automatically prints the result. When there is a single result to show, the print command is not needed.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the concat function to create two oml.DataFrame objects and combine the objects by column. The command automatically prints the result. When there is a single result to show, the print command is not needed.
.\n
An oml.Float is a numeric series data class that represents a single column of NUMBER, BINARY_DOUBLE, or BINARY_FLOAT database data types. Let's create an oml.Float object with the rounded exponential of two times the values in the num column of the oml.frame object, then concatenate it with the oml.DataFrame object y using a new column name.
.\n
Concatenate object x with multiple objects using automatic name conflict resolution. In this example, auto_name=True controls whether to call automatic name conflict resolution if one or more column names are duplicates in the two data frames. In this example, the id and num columns are duplicated.
.\n
Now concatenate multiple OML data objects with customized column renaming. We prepend the word New to the duplicate columns and use OrderedDict to preserve the order in which the objects are added.
Use the merge function to peform a cross join and a left outer join from two objects. This example first performs a cross join on the oml.DataFrame objects x and y, which creates the oml.DataFrame object xy.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-373210937","id":"20210302-002154_824808736","dateCreated":"2021-01-08T23:52:20+0000","dateStarted":"2021-03-02T18:55:37+0000","dateFinished":"2021-03-02T18:55:37+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:73"},{"text":"%python\n\nx = MY_DF[['id', 'val']]\ny = MY_DF[['num', 'ch']]\n\nxy = x.merge(y)\nxy\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:37+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" id_l val_l num_r ch_r\n0 1 a 4.0 p\n1 2 b 4.0 p\n2 3 c 4.0 p\n3 4 d 4.0 p\n4 5 e 4.0 p\n5 1 a 3.0 q\n6 2 b 3.0 q\n7 3 c 3.0 q\n8 4 d 3.0 q\n9 5 e 3.0 q\n10 1 a 6.7 r\n11 2 b 6.7 r\n12 3 c 6.7 r\n13 4 d 6.7 r\n14 5 e 6.7 r\n15 1 a 7.2 a\n16 2 b 7.2 a\n17 3 c 7.2 a\n18 4 d 7.2 a\n19 5 e 7.2 a\n20 1 a 5.0 b\n21 2 b 5.0 b\n22 3 c 5.0 b\n23 4 d 5.0 b\n24 5 e 5.0 b\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-1402248331","id":"20210302-002154_549002579","dateCreated":"2021-01-09T00:07:04+0000","dateStarted":"2021-03-02T18:55:38+0000","dateFinished":"2021-03-02T18:55:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:74"},{"text":"%md\n\n.\nPerform a left outer join on the first four rows of x with the oml.DataFrame object other on the shared column id and applies the suffixes *.l* and *.r* to column names on the left and right side, respectively.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:38+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Perform a left outer join on the first four rows of x with the oml.DataFrame object other on the shared column id and applies the suffixes .l and .r to column names on the left and right side, respectively.
Using the merge help file as a guide, perform a merge with a right outer join on the id column on the left side object x and the num column on the right side object y.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-2040759965","id":"20210302-002154_1670753656","dateCreated":"2021-01-09T01:09:03+0000","dateStarted":"2021-03-02T18:55:40+0000","dateFinished":"2021-03-02T18:55:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:77"},{"text":"%python\n\nhelp(oml.DataFrame.merge)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on cython_function_or_method in module oml.core.frame:\n\nmerge(self, other, on=None, left_on=None, right_on=None, how='left', suffixes=('_l', '_r'), nvl=True)\n Joins data sets.\n \n Parameters\n ----------\n other : an OML data set object\n on : str or list of str, optional\n Column names to join on. Must be found in both ``self`` and ``other``.\n left_on : str or list of str, optional\n Column names of ``self`` to join on.\n right_on : str or list of str, optional\n Column names of ``other`` to join on. If specified, must specify the same\n number of columns as ``left_on``.\n how : 'left' (default), 'right', 'inner', 'full'\n * left : left outer join\n * right : right outer join\n * full : full outer join\n * inner : inner join\n \n If ``on`` and ``left_on`` are both None, then ``how`` is ignored,\n and a cross join is performed.\n suffixes : sequence of length 2\n Suffix to apply to column names on the left and right side,\n respectively.\n nvl : True (default), False, dict \n * True : join condition includes NULL value\n * False : join condition excludes NULL value\n * dict : specifies the values that join columns use in replacement of NULL value with column names as keys\n \n Returns\n -------\n merged : oml.DataFrame\n\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-810288620","id":"20210302-002154_195979228","dateCreated":"2021-01-09T00:14:12+0000","dateStarted":"2021-03-02T18:55:40+0000","dateFinished":"2021-03-02T18:55:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:78"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644514465_1117596516","id":"20210302-002154_2030642284","dateCreated":"2021-01-10T17:43:32+0000","dateStarted":"2021-03-02T18:55:41+0000","dateFinished":"2021-03-02T18:55:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:79"},{"title":"Drop rows and columns","text":"%md\n\nIn preparing data for analysis, a typical step is to transform data by dropping some values. You can filter out data that are not needed by using the ***drop***, ***drop_duplicates***, and ***dropna*** methods. \n\nUse the ***del*** statement to remove an OML DataFrame proxy object and its associated temporary table. Note that ***del*** does not delete a persistent table. Use the ***oml.drop*** function to delete a persistent database table. To work with the ***drop*** method, first create a demo data table MY_DF2. Create a temporary table MY_DF2 from Pandas dataframe.\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"In preparing data for analysis, a typical step is to transform data by dropping some values. You can filter out data that are not needed by using the drop, drop_duplicates, and dropna methods.
\nUse the del statement to remove an OML DataFrame proxy object and its associated temporary table. Note that del does not delete a persistent table. Use the oml.drop function to delete a persistent database table. To work with the drop method, first create a demo data table MY_DF2. Create a temporary table MY_DF2 from Pandas dataframe.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_769615273","id":"20210302-002154_585277670","dateCreated":"2021-01-09T00:48:50+0000","dateStarted":"2021-03-02T18:55:42+0000","dateFinished":"2021-03-02T18:55:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:80"},{"text":"%python\n\nmy_df2 = pd.DataFrame({'numeric': [1, 1.4, -4, -4, 5.432, None, None],\n 'string1': [None, None, 'a', 'a', 'a', 'b', None],\n 'string2': ['x', None, 'z', 'z', 'z', 'x', None]})\n\n\nMY_DF2 = oml.push(my_df2, dbtypes = {'numeric': 'BINARY_DOUBLE', \n 'string1':'CHAR(1)', \n 'string2':'CHAR(1)'})\n\nMY_DF2\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:42+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" numeric string1 string2\n0 1.400 None None\n1 NaN None None\n2 1.000 None x\n3 NaN b x\n4 -4.000 a z\n5 -4.000 a z\n6 5.432 a z\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_-657487995","id":"20210302-002154_1817327535","dateCreated":"2021-01-09T19:46:06+0000","dateStarted":"2021-03-02T18:55:42+0000","dateFinished":"2021-03-02T18:55:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:81"},{"text":"%md\n\n.\nDrop rows with missing values.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:55:43+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Drop rows with missing values.
.\n
Drop rows with missing numeric values.
.\n
Drop rows where all columns values are missing.
.\n
Use the drop_duplicates function to drop duplicate rows.
.\n
The following command drops the specified column, string2.
This section demonstrates how to use the split and KFold functions using the digits data set after creating an OML DataFrame proxy object for the digits data set.
\nThe KFold method splits the series data object randomly into k consecutive folds, or non-overlapping groups, for use with k-fold cross validation. The split method splits the series data object randomly into multiple sets.
\nThe following tasks are covered in this section
\n.\n
First, use oml.sync to create a DataFrame proxy object in Python that represents the DIGITS table. The DIGITS table resides in the OMLUSER schema. All database users have been granted read access on the table.
.\n
Split the data set into samples of 80% and 20% size.
.\n
Perform stratified sampling on the column, target. Stratified sampling divides the data into different groups based on shared characteristics prior to sampling, ensuring that members from each subgroup is included in the analysis.
.\n
Verify that the stratified sampling generates splits in which all of the different categories of digits (digits 0~9) are present in each split.
.\n
Compute hash on the target column.
.\n
Verify that the different categories of digits (digits 0~9) are present in only one of the splits generated by hashing on the category column.
.\n
Split the data randomly into 4 consecutive folds using the KFold function.
Split the digit data set into four samples of 25% each.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_154485003","id":"20210302-002154_786656985","dateCreated":"2021-01-09T20:21:10+0000","dateStarted":"2021-03-02T18:56:02+0000","dateFinished":"2021-03-02T18:56:02+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:107"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:02+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644514465_-555226593","id":"20210302-002154_350992421","dateCreated":"2021-01-10T17:28:15+0000","dateStarted":"2021-03-02T18:56:02+0000","dateFinished":"2021-03-02T18:56:02+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:108"},{"title":"Step 6: Use the crosstab and pivot_table functions","text":"%md\n\nThis step shows how to use the ***crosstab*** method to perform cross-column analysis of an oml.DataFrame object and the ***pivot_table*** method to convert an oml.DataFrame to a spreadsheet-style pivot table.\n\nCross-tabulation is a statistical technique that finds an interdependent relationship between two columns of values. The ***crosstab*** method computes a cross-tabulation of two or more columns. By default, it computes a frequency table for the columns unless a column and an aggregation function have been passed to it.\n\nThe ***pivot_table*** function converts a data set into a pivot table. Due to the database 1000 column limit, pivot tables with more than 1000 columns are automatically truncated to display the categories with the most entries for each column value.\n\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:02+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"This step shows how to use the crosstab method to perform cross-column analysis of an oml.DataFrame object and the pivot_table method to convert an oml.DataFrame to a spreadsheet-style pivot table.
\nCross-tabulation is a statistical technique that finds an interdependent relationship between two columns of values. The crosstab method computes a cross-tabulation of two or more columns. By default, it computes a frequency table for the columns unless a column and an aggregation function have been passed to it.
\nThe pivot_table function converts a data set into a pivot table. Due to the database 1000 column limit, pivot tables with more than 1000 columns are automatically truncated to display the categories with the most entries for each column value.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514465_307729424","id":"20210302-002154_1025172074","dateCreated":"2021-01-09T20:22:18+0000","dateStarted":"2021-03-02T18:56:03+0000","dateFinished":"2021-03-02T18:56:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:109"},{"text":"%md\n\n.\nCreate a temporary table *MY_DF4* using a demo data set. This data set reports on the speed and accuracy for a given task characterized by hand used (left or right) and gender (male, female, or not specified).\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:03+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create a temporary table MY_DF4 using a demo data set. This data set reports on the speed and accuracy for a given task characterized by hand used (left or right) and gender (male, female, or not specified).
.\n
Use the crosstab function to find the categories that the most entries belonged to. This example shows how to find the count of gender, and right-handed and left-handed persons in descending order in the MY_DF4 dataframe.
.\n
Use the crosstab function to find the ratio of entries with different hand values for each gender and across entries in the MY_DF4 dataframe.
.\n
Use the pivot_table function to find the mean speed across all gender and hand combinations in the MY_DF4 dataframe.
.\n
Use the pivot_table function to find the maximum and minimum speed for every gender and hand combination and across all combinations in the MY_DF4 dataframe.
OML4Py provides functions for rendering graphical displays of data. The oml.boxplot and oml.hist functions compute the statistics necessary to generate box and whisker plots or histograms in-database for scalability and performance. OML4Py uses the matplotlib library to render the output.
\nThis example demonstrates how to use the oml.boxplot and oml.hist functions using the wine data set. The statistics supporting these plots are computed in the database, so transfer of data or client side memory limitations are avoided. As before, we sync the WINE table in the OMLUSER schema to a local OML DataFrame.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_1419080432","id":"20210302-002154_50842053","dateCreated":"2021-01-09T20:44:55+0000","dateStarted":"2021-03-02T18:56:12+0000","dateFinished":"2021-03-02T18:56:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:120"},{"title":"Create a boxplot","text":"%python\n\nimport matplotlib.pyplot as plt\n\nWINE = oml.sync(table = \"WINE\")\n\noml.graphics.boxplot(WINE[:,8:12], showmeans=True, meanline=True, patch_artist=True, labels=WINE.columns[8:12])\nplt.title('Distribution of Wine Attributes')\nplt.ylabel('Attribute Value')\nplt.show()\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:12+0000","config":{"colWidth":6,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true,"editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"
Use the oml.create function to create a persistent table in your database schema from data in your Python session. The oml.create function creates a table in the database schema and returns an oml.DataFrame object that is a proxy for the table. The proxy oml.DataFrame object has the same name as the table. The resulting table is available from both Python and SQL, even after the session ends - in contrast to the temporary table created when using the oml.push function
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_-657681706","id":"20210302-002154_1396176212","dateCreated":"2021-01-09T20:50:22+0000","dateStarted":"2021-03-02T18:56:21+0000","dateFinished":"2021-03-02T18:56:21+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:123"},{"text":"%python\n\ntry:\n oml.drop(table=\"IRIS\")\nexcept:\n pass\n\nIRIS = oml.create(iris_df, table=\"IRIS\")\nprint(\"Shape:\",IRIS.shape)\nz.show(IRIS.head(10))\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:21+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"1":{"graph":{"mode":"table","height":300,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"SEPAL_LENGTH":"string","SEPAL_WIDTH":"string","PETAL_LENGTH":"string","PETAL_WIDTH":"string","SPECIES":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Shape: (150, 5)\n"},{"type":"TABLE","data":"SEPAL_LENGTH\tSEPAL_WIDTH\tPETAL_LENGTH\tPETAL_WIDTH\tSPECIES\n4.6\t3.6\t1\t0.2\tsetosa\n5.1\t2.5\t3\t1.1\tversicolor\n6.0\t2.2\t4\t1.0\tversicolor\n5.8\t2.6\t4\t1.2\tversicolor\n5.5\t2.3\t4\t1.3\tversicolor\n5.5\t2.5\t4\t1.3\tversicolor\n6.1\t2.8\t4\t1.3\tversicolor\n5.7\t2.5\t5\t2.0\tvirginica\n6.0\t2.2\t5\t1.5\tvirginica\n6.3\t2.5\t5\t1.9\tvirginica\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_-1612983924","id":"20210302-002154_1849902870","dateCreated":"2021-01-10T17:58:51+0000","dateStarted":"2021-03-02T18:56:22+0000","dateFinished":"2021-03-02T18:56:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:124"},{"title":"View persistent tables in the user schema","text":"%python\n\nlist(oml.sync(table=\".*\", regex_match=True).keys())","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:24+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined","title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"['IRIS', 'TEST_DATA', 'WINE', 'DIGITS', 'IRIS_ID']\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_-785033937","id":"20210302-002154_1566464585","dateCreated":"2021-01-14T20:38:36+0000","dateStarted":"2021-03-02T18:56:24+0000","dateFinished":"2021-03-02T18:56:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:125"},{"text":"%md\n\n.\nCreate a cursor object for the database connection using ***oml.cursor*** from **cx_Oracle**, then shows the SQL data types of the columns in table *IRIS*.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:25+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create a cursor object for the database connection using oml.cursor from cx_Oracle, then shows the SQL data types of the columns in table IRIS.
.\n
To close the cursor, run the cr.close command.
.\n
Use the transparency layer function shape to view the shape, or number of rows and columns, of the IRIS table.
.\n
Use the transparency layer function head to view the first three rows of the IRIS table.
Use describe to calculate descriptive statistics that summarize the central tendency, dispersion, and shape of the IRIS table in each numeric column.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_455999290","id":"20210302-002154_1661624158","dateCreated":"2021-01-10T18:35:55+0000","dateStarted":"2021-03-02T18:56:30+0000","dateFinished":"2021-03-02T18:56:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:134"},{"text":"%python\n\nIRIS.describe()\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" SEPAL_LENGTH SEPAL_WIDTH PETAL_LENGTH PETAL_WIDTH\ncount 150.000000 150.000000 150.000000 150.000000\nmean 5.843333 3.057333 3.758000 1.199333\nstd 0.828066 0.435866 1.765298 0.762238\nmin 4.300000 2.000000 1.000000 0.100000\n25% 5.100000 2.800000 1.600000 0.300000\n50% 5.800000 3.000000 4.350000 1.300000\n75% 6.400000 3.300000 5.100000 1.800000\nmax 7.900000 4.400000 6.900000 2.500000\n"}]},"interrupted":false,"jobName":"paragraph_1614644514466_-2064513617","id":"20210302-002154_1614835265","dateCreated":"2021-01-10T18:39:20+0000","dateStarted":"2021-03-02T18:56:31+0000","dateFinished":"2021-03-02T18:56:31+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:135"},{"text":"%md\n\n## End of Script\n","user":"OMLUSER02","dateUpdated":"2021-03-02T18:56:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis lab highlights a few of the machine learning algorithms available in OML4Py: generalized linear models, K-Means Clustering, partitioned models, and model explainability.
\nCopyright (c) 2021 Oracle Corporation
\nThis example shows how to predict numerical values using multiple regression. Given demographic, purchase, and affinity card data for a set of customers, predict the number of years a customer remains at the same residence, as found in column YRS_RESIDENCE - a continuous variable. We will use the Generalized Linear Model algorithm. All processing occurs inside Oracle Autonomous Database.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-1711631848","id":"20210125-170341_725904389","dateCreated":"2021-01-12T21:16:55+0000","dateStarted":"2021-02-06T19:25:33+0000","dateFinished":"2021-02-06T19:25:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"text":"%md\n. \n\nPrepare the dataset combining the *CUSTOMERS* table from the SH schema with the *SUPPLEMENTARY_DEMOGRAPHICS* table. Note that we're illustrating how to use the query specification with ***oml.sync*** and selecting the columns we want to include. However, we could also have used ***oml.sync*** on the *CUSTOMERS* and *SUPPLEMENTARY_DEMOGRAPHICS* tables themselves, and perform column filtering in Python prior to the merge.\n\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:33+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".
\nPrepare the dataset combining the CUSTOMERS table from the SH schema with the SUPPLEMENTARY_DEMOGRAPHICS table. Note that we're illustrating how to use the query specification with oml.sync and selecting the columns we want to include. However, we could also have used oml.sync on the CUSTOMERS and SUPPLEMENTARY_DEMOGRAPHICS tables themselves, and perform column filtering in Python prior to the merge.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_1598126648","id":"20210125-170341_833674782","dateCreated":"2021-01-12T21:18:26+0000","dateStarted":"2021-02-06T19:25:34+0000","dateFinished":"2021-02-06T19:25:34+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:43"},{"text":"%python\n\nCUSTOMERS = oml.sync(query = 'SELECT CUST_ID, CUST_GENDER, CUST_MARITAL_STATUS, CUST_YEAR_OF_BIRTH, CUST_INCOME_LEVEL, CUST_CREDIT_LIMIT FROM SH.CUSTOMERS')\nDEMO_DF = oml.sync(query = \"\"\"SELECT CUST_ID, EDUCATION, AFFINITY_CARD, HOUSEHOLD_SIZE, OCCUPATION, YRS_RESIDENCE, Y_BOX_GAMES FROM SH.SUPPLEMENTARY_DEMOGRAPHICS\"\"\")\nCUST_DF = CUSTOMERS.merge(DEMO_DF, how = \"inner\", on = 'CUST_ID',suffixes = [\"\",\"\"])\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:34+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594221794_-397328202","id":"20210125-170341_1874600151","dateCreated":"2021-01-12T21:18:53+0000","dateStarted":"2021-02-06T19:25:34+0000","dateFinished":"2021-02-06T19:25:35+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:44"},{"text":"%md\n.\nDisplay the first few rows of table *CUST_DF*.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:35+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display the first few rows of table CUST_DF.
.\n
Split data into randomly selected 60% sample for train and 40% test.
.\n
Build a GLM regression model for predicting YRS_RESIDENCE using the function oml.glm.
This method runs the in-database oml.glm algorithm in-database using the provided settings. The settings are supplied as key-value pairs. In this example we specify feature generation and feature selection. For more information, refer to the oml.glm help file.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_1501168440","id":"20210125-170341_32469475","dateCreated":"2021-01-12T21:20:52+0000","dateStarted":"2021-02-06T19:25:38+0000","dateFinished":"2021-02-06T19:25:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:49"},{"text":"%python\n\nhelp(oml.glm)\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:38+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on class glm in module oml.algo.glm:\n\nclass glm(oml.algo.model.odmModel)\n | glm(mining_function='CLASSIFICATION', model_name=None, model_owner=None, **params)\n | \n | In-database `Generalized Linear Models.
\nDisplay model fit details to understand the model's key statistics. Locate the values of Root Mean Square Error ROOT_MEAN_SQ and R-squared R_SQ from the output. RMSE and R-squared are used to evaluate baseline performance of the model.
\nRMSE is a measure of the differences between values predicted by a model and the values observed. A good model should have a low RMSE. But at the same time, a model with very low RMSE has the potential to overfit.
\nR-Squared is a statistical measure that represents the goodness of fit of a regression model. The ideal value for R-squared is 1. The closer the value of R-squared is to 1, the better the model fit. For example, if the R-squared of a model is 0.50, then only about half of the observed variation can be explained by the model's inputs.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-975928578","id":"20210125-170341_1987738175","dateCreated":"2021-01-12T21:21:40+0000","dateStarted":"2021-02-06T19:25:45+0000","dateFinished":"2021-02-06T19:25:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:52"},{"text":"%python\n\nz.show(glm_mod.fit_details)\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:45+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":300,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"name":"string","value":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"name\tvalue\nADJUSTED_R_SQUARE\t0.5093743672999828\nAIC\t1502.8033446013962\nCOEFF_VAR\t32.77662502584026\nCORRECTED_TOTAL_DF\t2708.0\nCORRECTED_TOT_SS\t9534.730897009955\nDEPENDENT_MEAN\t4.009966777408638\nERROR_DF\t2687.0\nERROR_MEAN_SQUARE\t1.727468012913557\nERROR_SUM_SQUARES\t4641.706550698727\nF_VALUE\t134.8801584784037\nGMSEP\t1.7416165687323748\nHOCKING_SP\t0.0006431377561107807\nJ_P\t1.7414969151963544\nMODEL_CONVERGED\t1.0\nMODEL_DF\t21.0\nMODEL_F_P_VALUE\t0.0\nMODEL_MEAN_SQUARE\t233.0011593481537\nMODEL_SUM_SQUARES\t4893.024346311227\nNUM_PARAMS\t22.0\nNUM_ROWS\t2709.0\nROOT_MEAN_SQ\t1.314331774292\nR_SQ\t0.5131790712463271\nSBIC\t1632.6987111272683\nVALID_COVARIANCE_MATRIX\t1.0\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-1040860550","id":"20210125-170341_207650170","dateCreated":"2021-01-12T21:22:07+0000","dateStarted":"2021-02-06T19:25:45+0000","dateFinished":"2021-02-06T19:25:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"text":"%md\n\n.\nDisplay model coefficients to estimate the relationship between the predictor and response variables.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:46+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display model coefficients to estimate the relationship between the predictor and response variables.
.\n
Make predictions using test data and display results.
.\n
Plot the predicted versus actual years of residence. Click on settings to see how the plot was specified.
.\n
Using matplotlib, plot the predicted and actual years of residence.
.\n
Plot the residuals using matplotlib.
.
\nCompute RMSE explicitly using the prediction results. Also compute R-squared using the test data set using the model's score method. Note that both the RMSE and R-squared are similar to the values produced automatically by oml.glm.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_382433762","id":"20210125-170341_2072601514","dateCreated":"2021-01-12T21:46:38+0000","dateStarted":"2021-02-06T19:25:55+0000","dateFinished":"2021-02-06T19:25:55+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:64"},{"text":"%python\n\nprint(((RES_DF['YRS_RESIDENCE'] - RES_DF['PREDICTION']) ** 2).mean() ** .5)\n\nprint(glm_mod.score(TEST.drop('YRS_RESIDENCE'), TEST[:,['YRS_RESIDENCE']]))\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:56+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"1.402635365872594\n0.477143\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_1751201584","id":"20210125-170341_682784499","dateCreated":"2021-01-12T22:00:35+0000","dateStarted":"2021-02-06T19:25:56+0000","dateFinished":"2021-02-06T19:25:57+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:65"},{"title":"Step 3: Clustering using K-Means","text":"%md\n\nOML4Py supports clustering using several algorithms: K-Means, O-Cluster, and Expectation Maximization. In this lab, we illustrate how to identify natural clusters of customers using the *CUSTOMERS* dataset and the unsupervised learning K-Means algorithm. Note that data exploration, preparation, and machine learning run inside Autonomous Database.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:57+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"OML4Py supports clustering using several algorithms: K-Means, O-Cluster, and Expectation Maximization. In this lab, we illustrate how to identify natural clusters of customers using the CUSTOMERS dataset and the unsupervised learning K-Means algorithm. Note that data exploration, preparation, and machine learning run inside Autonomous Database.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_1920877208","id":"20210125-170341_730251635","dateCreated":"2021-01-12T22:03:34+0000","dateStarted":"2021-02-06T19:25:57+0000","dateFinished":"2021-02-06T19:25:57+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:66"},{"text":"%md\n\n.\nBuild a *k-Means* clustering model with 3 clusters, first dropping the model *CUST_CLUSTER_MODEL* if it exists. Specify the number of iterations for building the cluster tree using the *KMNS_ITERATIONS* setting and the K-Means random seed generator using the *KMNS_RANDOM_SEED* setting. *KMNS_ITERATIONS* specifies the number of times the algorithm should iterate over the training data before it finalizes the selection of centroids. You can adjust this parameter to balance accuracy against training time. The *KMNS_RANDOM_SEED* value is used for cluster initialization and can have a significant effect on cluster selection.\n\nNote that *model_name* causes the model to be persisted across notebook sessions.","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:57+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Build a k-Means clustering model with 3 clusters, first dropping the model CUST_CLUSTER_MODEL if it exists. Specify the number of iterations for building the cluster tree using the KMNS_ITERATIONS setting and the K-Means random seed generator using the KMNS_RANDOM_SEED setting. KMNS_ITERATIONS specifies the number of times the algorithm should iterate over the training data before it finalizes the selection of centroids. You can adjust this parameter to balance accuracy against training time. The KMNS_RANDOM_SEED value is used for cluster initialization and can have a significant effect on cluster selection.
Note that model_name causes the model to be persisted across notebook sessions.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-2082300111","id":"20210125-170341_383774044","dateCreated":"2021-01-12T22:34:24+0000","dateStarted":"2021-02-06T19:25:58+0000","dateFinished":"2021-02-06T19:25:58+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:67"},{"text":"%python\n\ntry:\n oml.drop(model=\"CUST_CLUSTER_MODEL\")\nexcept:\n print(\"Model does not exist\")\n\nsetting = {'KMNS_ITERATIONS': 20, 'KMNS_RANDOM_SEED': 7}\nkm_mod = oml.km(n_clusters = 3, **setting).fit(CUST_DF, model_name = \"CUST_CLUSTER_MODEL\", case_id = 'CUST_ID')","user":"OMLUSER03","dateUpdated":"2021-02-06T19:25:58+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594221794_-2001724888","id":"20210125-170341_645617694","dateCreated":"2021-01-12T22:31:33+0000","dateStarted":"2021-02-06T19:25:58+0000","dateFinished":"2021-02-06T19:26:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:68"},{"text":"%md\n\n.\nDisplay the K-Means model details. \n\n ","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:03+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display the K-Means model details.
.\n
Display the cluster details for all clusters in the hierarchy with row counts and dispersion. The dispersion value is a measure of how compact or how spread out the data is within a cluster. The dispersion value is greater than 0 - the lower the dispersion value, the more compact the cluster, and the closer the data points are to the centroid of the cluster. A larger value indicates the data points are more disperse or spread out from the controid.
.\n
The taxonomy shows the hierarchy of the child clusters in relationship to the parent clusters.
.\n
Use the model to predict cluster membership. The supplemental_cols argument carries the target column to the output to retain the relationship between the predictions and their original preditor values. These predictors may include a case id, for example to join with other tables, or mutiple (or all) columns of the scoring data. Users should be aware that unlike Pandas DataFrames, which are explicitly ordered in memory, results from relational databases do not have a specific order unless explicitly specified by an ORDER BY clause. As such, users cannot rely on results to maintain the same order across different data sets (tables and DataFrame proxy objects).
.\n
Finally, view the cluster results using a matplotlib scatterplot. We've selected two predictors to show cluster relationships in two dimensions: YRS_RESIDENCE and CUST_YEAR_OF_BIRTH.
OML4Py enables automatically building an ensemble model comprised of multiple sub-models, one for each data partition. Sub-models exist and are used as one model, which results in simplified scoring using the top-level model only. The proper sub-model is chosen by the system based on partition value(s) in the row of data to be scored. Partitioned models achieve potentially better accuracy through multiple targeted models.
\nIn this lab, we build an SVM model to predict the number of years a customer resides at their residence but partitioned on customer gender. The model is then used to predict the target, then predict the target with prediction details.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_1530518706","id":"20210125-170341_702035530","dateCreated":"2021-01-12T22:09:32+0000","dateStarted":"2021-02-06T19:26:14+0000","dateFinished":"2021-02-06T19:26:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:80"},{"text":"%md\n\n.\nCreate test and train data sets and drop missing values.","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:14+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create test and train data sets and drop missing values.
.\n
Build Partitioned Model using the SVM algorithm to predict YRS_RESIDENCE, partitioned by CUST_GENDER.
.\n
Predict on the test set and display prediction result - note the use of the top level model only.
.\n
Show model global statistics for each partitioned sub-model. The partition name column contains the values from the partition column. If multiple columns were specified, there would be one column for each with corresponding value.
Materialize the data to table TEST_DATA so it can be queried from SQL.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_779144562","id":"20210125-170341_1350990915","dateCreated":"2021-01-13T00:58:14+0000","dateStarted":"2021-02-06T19:26:23+0000","dateFinished":"2021-02-06T19:26:23+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:89"},{"text":"%python\n\ntry:\n oml.drop(table = 'TEST_DATA')\nexcept:\n print(\"Table not exist\")\n_ = test_dat.materialize(table = 'TEST_DATA')\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:23+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594221794_-282402079","id":"20210125-170341_1849913545","dateCreated":"2021-01-13T00:58:45+0000","dateStarted":"2021-02-06T19:26:24+0000","dateFinished":"2021-02-06T19:26:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:90"},{"text":"%md\n\n.\nUse SQL to score data and display prediction details.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:24+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use SQL to score data and display prediction details.
Machine Learning Explainability, MLX, supports explaining and interpreting machine learning models. The oml.mlx module enables understanding a model's behavior and how it ranks predictors when making predictions in an algorithm-agnostic way. MLX currently provides these model-agnostic explanations for classification and regression models where explanations treat the ML model as a black-box, instead of using properties from the model to guide the explanation.
\nThe Global Feature Importance explainer object is the interface to the MLX permutation importance explainer. It identifies the most important features for a given model and data set. The explainer is model-agnostic and currently supports tabular classification and regression data with both numerical and categorical features.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_530692173","id":"20210125-170341_1586756635","dateCreated":"2021-01-12T22:22:53+0000","dateStarted":"2021-02-06T19:26:41+0000","dateFinished":"2021-02-06T19:26:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:93"},{"text":"%md\n\n.\n\nImport the *GlobalFeatureImportance* explainer, and load the iris data set into the database. The script also adds a unique Case ID column.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".
\nImport the GlobalFeatureImportance explainer, and load the iris data set into the database. The script also adds a unique Case ID column.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-2098110145","id":"20210125-170341_732109368","dateCreated":"2021-01-12T22:23:56+0000","dateStarted":"2021-02-06T19:26:42+0000","dateFinished":"2021-02-06T19:26:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:94"},{"text":"%python\n\nfrom oml.mlx import GlobalFeatureImportance\nfrom sklearn import datasets\n\niris_ds = datasets.load_iris()\niris_data = iris_ds.data.astype(float)\nX = pd.DataFrame(iris_data, columns=iris_ds.feature_names)\ny = pd.DataFrame(iris_ds.target, columns=['TARGET'])\nrow_id = pd.DataFrame(np.arange(iris_data.shape[0]), \ncolumns=['CASE_ID'])\n\n\ntry:\n oml.drop(table=\"IRIS_ID\")\nexcept:\n pass\ndf = oml.create(pd.concat([X, y, row_id], axis=1), table='IRIS_ID')\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:42+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594221794_243763378","id":"20210125-170341_582040478","dateCreated":"2021-01-13T01:09:25+0000","dateStarted":"2021-02-06T19:26:42+0000","dateFinished":"2021-02-06T19:26:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:95"},{"text":"%md\n\n.\nSplit the data set into train and test sets.\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:43+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Split the data set into train and test sets.
.\n
Train an SVM model.
.\n
Create the MLX Global Feature Importance explainer gfi, using the f1_weighted metric.
.
\nInvoke gfi.explain to generate the global feature importance for the provided data. The explainer returns an ordered list of features along with their importance values. Features located at higher ranks have more impact on the model predictions. Importance values also represent relative feature importance. Here we see that petal length is the most important feature.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-623339802","id":"20210125-170341_2091986805","dateCreated":"2021-01-13T01:22:40+0000","dateStarted":"2021-02-06T19:26:49+0000","dateFinished":"2021-02-06T19:26:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:102"},{"text":"%python\n\nexplanation = gfi.explain(model, X, y,\ncase_id='CASE_ID', n_iter=10)\nexplanation\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:49+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Global Feature Importance: \n\t[0] petal length (cm): Value: 0.3496, Error: 0.0236\n\t[1] petal width (cm): Value: 0.2440, Error: 0.0332\n\t[2] sepal width (cm): Value: 0.0861, Error: 0.0197\n\t[3] sepal length (cm): Value: 0.0108, Error: 0.0124\n\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_2138158495","id":"20210125-170341_748092270","dateCreated":"2021-01-13T01:22:58+0000","dateStarted":"2021-02-06T19:26:50+0000","dateFinished":"2021-02-06T19:26:53+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:103"},{"title":"Try it yourself","text":"%md\nBuild an in-database RandomForest model and compare the model's attribute importance ranking with that from *MLX*.","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:53+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","title":true,"editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Build an in-database RandomForest model and compare the model's attribute importance ranking with that from MLX.
\n"}]},"interrupted":false,"jobName":"paragraph_1611594221794_-736684108","id":"20210125-170341_1802202519","dateCreated":"2021-01-20T05:10:19+0000","dateStarted":"2021-02-06T19:26:54+0000","dateFinished":"2021-02-06T19:26:54+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:104"},{"text":"%python\n\n# Insert code here\n\n\n\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:54+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1611594221794_1919367248","id":"20210125-170341_1795203183","dateCreated":"2021-01-20T05:11:32+0000","dateStarted":"2021-02-06T19:26:54+0000","dateFinished":"2021-02-06T19:26:54+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:105"},{"text":"%md\n\n# End of Script\n","user":"OMLUSER03","dateUpdated":"2021-02-06T19:26:55+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis lab highlights a few of the machine learning algorithms and features available in OML4Py: generalized linear models, K-Means Clustering, partitioned models, and model explainability.
\nCopyright (c) 2021 Oracle Corporation
\nThis example shows how to predict numerical values using multiple regression. Given demographic, purchase, and affinity card data for a set of customers, predict the number of years a customer remains at the same residence, as found in column YRS_RESIDENCE - a continuous variable. We will use the Generalized Linear Model algorithm. All processing occurs inside Oracle Autonomous Database.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_697098812","id":"20210302-002652_496445347","dateCreated":"2021-01-12T21:16:55+0000","dateStarted":"2021-03-02T21:13:31+0000","dateFinished":"2021-03-02T21:13:31+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"text":"%md\n. \n\nPrepare the dataset combining the *CUSTOMERS* table from the SH schema with the *SUPPLEMENTARY_DEMOGRAPHICS* table. Note that we're illustrating how to use the query specification with ***oml.sync*** and selecting the columns we want to include. However, we could also have used ***oml.sync*** on the *CUSTOMERS* and *SUPPLEMENTARY_DEMOGRAPHICS* tables themselves, and perform column filtering in Python prior to the merge.\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".
\nPrepare the dataset combining the CUSTOMERS table from the SH schema with the SUPPLEMENTARY_DEMOGRAPHICS table. Note that we're illustrating how to use the query specification with oml.sync and selecting the columns we want to include. However, we could also have used oml.sync on the CUSTOMERS and SUPPLEMENTARY_DEMOGRAPHICS tables themselves, and perform column filtering in Python prior to the merge.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-2026101568","id":"20210302-002652_750275638","dateCreated":"2021-01-12T21:18:26+0000","dateStarted":"2021-03-02T21:13:32+0000","dateFinished":"2021-03-02T21:13:32+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:43"},{"text":"%python\n\nCUSTOMERS = oml.sync(query = 'SELECT CUST_ID, CUST_GENDER, CUST_MARITAL_STATUS, CUST_YEAR_OF_BIRTH, CUST_INCOME_LEVEL, CUST_CREDIT_LIMIT FROM SH.CUSTOMERS')\nDEMO_DF = oml.sync(query = \"\"\"SELECT CUST_ID, EDUCATION, AFFINITY_CARD, HOUSEHOLD_SIZE, OCCUPATION, YRS_RESIDENCE, Y_BOX_GAMES FROM SH.SUPPLEMENTARY_DEMOGRAPHICS\"\"\")\nCUST_DF = CUSTOMERS.merge(DEMO_DF, how = \"inner\", on = 'CUST_ID',suffixes = [\"\",\"\"])\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:32+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812550_1079853580","id":"20210302-002652_606499330","dateCreated":"2021-01-12T21:18:53+0000","dateStarted":"2021-03-02T21:13:32+0000","dateFinished":"2021-03-02T21:13:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:44"},{"text":"%md\n.\nDisplay the first few rows of table *CUST_DF*.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:33+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display the first few rows of table CUST_DF.
.\n
Split data into randomly selected 60% sample for train and 40% test.
.\n
Build a GLM regression model for predicting YRS_RESIDENCE using the function oml.glm.
This method runs the in-database oml.glm algorithm in-database using the provided settings. The settings are supplied as key-value pairs. In this example we specify feature generation and feature selection. For more information, refer to the oml.glm help file.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-635905665","id":"20210302-002652_1948647535","dateCreated":"2021-01-12T21:20:52+0000","dateStarted":"2021-03-02T21:13:36+0000","dateFinished":"2021-03-02T21:13:36+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:49"},{"text":"%python\n\nhelp(oml.glm)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:36+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Help on class glm in module oml.algo.glm:\n\nclass glm(oml.algo.model.odmModel)\n | glm(mining_function='CLASSIFICATION', model_name=None, model_owner=None, **params)\n | \n | In-database `Generalized Linear Models.
\nDisplay model fit details to understand the model's key statistics. Locate the values of Root Mean Square Error ROOT_MEAN_SQ and R-squared R_SQ from the output. RMSE and R-squared are used to evaluate baseline performance of the model.
\nRMSE is a measure of the differences between values predicted by a model and the values observed. A good model should have a low RMSE. But at the same time, a model with very low RMSE has the potential to overfit.
\nR-Squared is a statistical measure that represents the goodness of fit of a regression model. The ideal value for R-squared is 1. The closer the value of R-squared is to 1, the better the model fit. For example, if the R-squared of a model is 0.50, then only about half of the observed variation can be explained by the model's inputs.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-1557741349","id":"20210302-002652_1680748890","dateCreated":"2021-01-12T21:21:40+0000","dateStarted":"2021-03-02T21:13:46+0000","dateFinished":"2021-03-02T21:13:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:52"},{"text":"%python\n\nz.show(glm_mod.fit_details)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:46+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":300,"optionOpen":false,"setting":{"table":{"tableGridState":{},"tableColumnTypeState":{"names":{"name":"string","value":"string"},"updated":false},"tableOptionSpecHash":"[{\"name\":\"useFilter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable filter for columns\"},{\"name\":\"showPagination\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable pagination for better navigation\"},{\"name\":\"showAggregationFooter\",\"valueType\":\"boolean\",\"defaultValue\":false,\"widget\":\"checkbox\",\"description\":\"Enable a footer for displaying aggregated values\"}]","tableOptionValue":{"useFilter":false,"showPagination":false,"showAggregationFooter":false},"updated":false,"initialized":false}},"commonSetting":{}}}},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"name\tvalue\nADJUSTED_R_SQUARE\t0.5093743672999828\nAIC\t1502.8033446013962\nCOEFF_VAR\t32.77662502584026\nCORRECTED_TOTAL_DF\t2708.0\nCORRECTED_TOT_SS\t9534.730897009955\nDEPENDENT_MEAN\t4.009966777408638\nERROR_DF\t2687.0\nERROR_MEAN_SQUARE\t1.727468012913557\nERROR_SUM_SQUARES\t4641.706550698727\nF_VALUE\t134.8801584784037\nGMSEP\t1.7416165687323748\nHOCKING_SP\t0.0006431377561107807\nJ_P\t1.7414969151963544\nMODEL_CONVERGED\t1.0\nMODEL_DF\t21.0\nMODEL_F_P_VALUE\t0.0\nMODEL_MEAN_SQUARE\t233.0011593481537\nMODEL_SUM_SQUARES\t4893.024346311227\nNUM_PARAMS\t22.0\nNUM_ROWS\t2709.0\nROOT_MEAN_SQ\t1.314331774292\nR_SQ\t0.5131790712463271\nSBIC\t1632.6987111272683\nVALID_COVARIANCE_MATRIX\t1.0\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_650380583","id":"20210302-002652_550465736","dateCreated":"2021-01-12T21:22:07+0000","dateStarted":"2021-03-02T21:13:47+0000","dateFinished":"2021-03-02T21:13:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"text":"%md\n\n.\nDisplay model coefficients to estimate the relationship between the predictor and response variables.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display model coefficients to estimate the relationship between the predictor and response variables.
.\n
Make predictions using test data and display results.
.\n
Plot the predicted versus actual years of residence. Click on settings to see how the plot was specified.
.\n
Using matplotlib, plot the predicted and actual years of residence and visually compare it against the perfect fit line, y=x.
.\n
Plot the residuals using matplotlib.
.
\nCompute RMSE explicitly using the prediction results. Also compute R-squared using the test data set using the model's score method. Note that both the RMSE and R-squared are similar to the values produced automatically by oml.glm.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_311068764","id":"20210302-002652_1460295364","dateCreated":"2021-01-12T21:46:38+0000","dateStarted":"2021-03-02T21:13:57+0000","dateFinished":"2021-03-02T21:13:57+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:64"},{"text":"%python\n\nprint(((RES_DF['YRS_RESIDENCE'] - RES_DF['PREDICTION']) ** 2).mean() ** .5)\n\nprint(glm_mod.score(TEST.drop('YRS_RESIDENCE'), TEST[:,['YRS_RESIDENCE']]))\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:58+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"1.402635365872594\n0.477143\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-1467370315","id":"20210302-002652_1174763936","dateCreated":"2021-01-12T22:00:35+0000","dateStarted":"2021-03-02T21:13:58+0000","dateFinished":"2021-03-02T21:13:59+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:65"},{"title":"Step 3: Clustering using K-Means","text":"%md\n\nOML4Py supports clustering using several algorithms: K-Means, O-Cluster, and Expectation Maximization. In this lab, we illustrate how to identify natural clusters of customers using the *CUSTOMERS* dataset and the unsupervised learning K-Means algorithm. Note that data exploration, preparation, and machine learning run inside Autonomous Database.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:59+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"OML4Py supports clustering using several algorithms: K-Means, O-Cluster, and Expectation Maximization. In this lab, we illustrate how to identify natural clusters of customers using the CUSTOMERS dataset and the unsupervised learning K-Means algorithm. Note that data exploration, preparation, and machine learning run inside Autonomous Database.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-926254290","id":"20210302-002652_917383659","dateCreated":"2021-01-12T22:03:34+0000","dateStarted":"2021-03-02T21:13:59+0000","dateFinished":"2021-03-02T21:13:59+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:66"},{"text":"%md\n\n.\nBuild a *k-Means* clustering model with 3 clusters, first dropping the model *CUST_CLUSTER_MODEL* if it exists. Specify the number of iterations for building the cluster tree using the *KMNS_ITERATIONS* setting and the K-Means random seed generator using the *KMNS_RANDOM_SEED* setting. *KMNS_ITERATIONS* specifies the number of times the algorithm should iterate over the training data before it finalizes the selection of centroids. You can adjust this parameter to balance accuracy against training time. The *KMNS_RANDOM_SEED* value is used for cluster initialization and can have a significant effect on cluster selection.\n\nNote that *model_name* causes the model to be persisted across notebook sessions.","user":"OMLUSER02","dateUpdated":"2021-03-02T21:13:59+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Build a k-Means clustering model with 3 clusters, first dropping the model CUST_CLUSTER_MODEL if it exists. Specify the number of iterations for building the cluster tree using the KMNS_ITERATIONS setting and the K-Means random seed generator using the KMNS_RANDOM_SEED setting. KMNS_ITERATIONS specifies the number of times the algorithm should iterate over the training data before it finalizes the selection of centroids. You can adjust this parameter to balance accuracy against training time. The KMNS_RANDOM_SEED value is used for cluster initialization and can have a significant effect on cluster selection.
Note that model_name causes the model to be persisted across notebook sessions.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1571950430","id":"20210302-002652_1870855429","dateCreated":"2021-01-12T22:34:24+0000","dateStarted":"2021-03-02T21:14:00+0000","dateFinished":"2021-03-02T21:14:00+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:67"},{"text":"%python\n\ntry:\n oml.drop(model=\"CUST_CLUSTER_MODEL\")\nexcept:\n print(\"Model does not exist\")\n\nsetting = {'KMNS_ITERATIONS': 20, 'KMNS_RANDOM_SEED': 7}\nkm_mod = oml.km(n_clusters = 3, **setting).fit(CUST_DF, model_name = \"CUST_CLUSTER_MODEL\", case_id = 'CUST_ID')","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:00+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812550_703174287","id":"20210302-002652_1790655047","dateCreated":"2021-01-12T22:31:33+0000","dateStarted":"2021-03-02T21:14:00+0000","dateFinished":"2021-03-02T21:14:07+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:68"},{"text":"%md\n\n.\nDisplay the K-Means model details. \n\n ","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:07+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Display the K-Means model details.
.\n
Display the cluster details for all clusters in the hierarchy with row counts and dispersion. The dispersion value is a measure of how compact or how spread out the data is within a cluster. The dispersion value is greater than 0 - the lower the dispersion value, the more compact the cluster, and the closer the data points are to the centroid of the cluster. A larger value indicates the data points are more disperse or spread out from the controid.
.\n
The taxonomy shows the hierarchy of the child clusters in relationship to the parent clusters.
.\n
Use the model to predict cluster membership. The supplemental_cols argument carries the target column to the output to retain the relationship between the predictions and their original preditor values. These predictors may include a case id, for example to join with other tables, or mutiple (or all) columns of the scoring data. Users should be aware that unlike Pandas DataFrames, which are explicitly ordered in memory, results from relational databases do not have a specific order unless explicitly specified by an ORDER BY clause. As such, users cannot rely on results to maintain the same order across different data sets (tables and DataFrame proxy objects).
.\n
Finally, view the cluster results using a matplotlib scatterplot. We've selected two predictors to show cluster relationships in two dimensions: YRS_RESIDENCE and CUST_YEAR_OF_BIRTH.
OML4Py enables automatically building an ensemble model comprised of multiple sub-models, one for each data partition. Sub-models exist and are used as one model, which results in simplified scoring using the top-level model only. The proper sub-model is chosen by the system based on partition value(s) in the row of data to be scored. Partitioned models achieve potentially better accuracy through multiple targeted models.
\nIn this lab, we build an SVM model to predict the number of years a customer resides at their residence but partitioned on customer gender. The model is then used to predict the target, then predict the target with prediction details.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_189431231","id":"20210302-002652_1441096651","dateCreated":"2021-01-12T22:09:32+0000","dateStarted":"2021-03-02T21:14:18+0000","dateFinished":"2021-03-02T21:14:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:80"},{"text":"%md\n\n.\nCreate test and train data sets and drop missing values.","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create test and train data sets and drop missing values.
.\n
Build Partitioned Model using the SVM algorithm to predict YRS_RESIDENCE, partitioned by CUST_GENDER.
.\n
Predict on the test set and display prediction result - note the use of the top level model only.
.\n
Show model global statistics for each partitioned sub-model. The partition name column contains the values from the partition column. If multiple columns were specified, there would be one column for each with corresponding value.
Materialize the data to table TEST_DATA so it can be queried from SQL.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1368161132","id":"20210302-002652_217068129","dateCreated":"2021-01-13T00:58:14+0000","dateStarted":"2021-03-02T21:14:28+0000","dateFinished":"2021-03-02T21:14:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:89"},{"text":"%python\n\ntry:\n oml.drop(table = 'TEST_DATA')\nexcept:\n print(\"Table not exist\")\n_ = test_dat.materialize(table = 'TEST_DATA')\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812550_-170558745","id":"20210302-002652_1781030599","dateCreated":"2021-01-13T00:58:45+0000","dateStarted":"2021-03-02T21:14:28+0000","dateFinished":"2021-03-02T21:14:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:90"},{"text":"%md\n\n.\nUse SQL to score data and display prediction details.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:29+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use SQL to score data and display prediction details.
Machine Learning Explainability, MLX, supports explaining and interpreting machine learning models. The oml.mlx module enables understanding a model's behavior and how it ranks predictors when making predictions in an algorithm-agnostic way. MLX currently provides these model-agnostic explanations for classification and regression models where explanations treat the ML model as a black-box, instead of using properties from the model to guide the explanation.
\nThe Global Feature Importance explainer object is the interface to the MLX permutation importance explainer. It identifies the most important features for a given model and data set. The explainer is model-agnostic and currently supports tabular classification and regression data with both numerical and categorical features.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1409456174","id":"20210302-002652_1981284239","dateCreated":"2021-01-12T22:22:53+0000","dateStarted":"2021-03-02T21:14:46+0000","dateFinished":"2021-03-02T21:14:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:93"},{"text":"%md\n\n.\n\nImport the *GlobalFeatureImportance* explainer, and load the iris data set into the database. The script also adds a unique Case ID column.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:46+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".
\nImport the GlobalFeatureImportance explainer, and load the iris data set into the database. The script also adds a unique Case ID column.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1451985725","id":"20210302-002652_1591826924","dateCreated":"2021-01-12T22:23:56+0000","dateStarted":"2021-03-02T21:14:47+0000","dateFinished":"2021-03-02T21:14:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:94"},{"text":"%python\n\nfrom oml.mlx import GlobalFeatureImportance\nfrom sklearn import datasets\n\niris_ds = datasets.load_iris()\niris_data = iris_ds.data.astype(float)\nX = pd.DataFrame(iris_data, columns=iris_ds.feature_names)\ny = pd.DataFrame(iris_ds.target, columns=['TARGET'])\nrow_id = pd.DataFrame(np.arange(iris_data.shape[0]), \ncolumns=['CASE_ID'])\n\n\ntry:\n oml.drop(table=\"IRIS_ID\")\nexcept:\n pass\ndf = oml.create(pd.concat([X, y, row_id], axis=1), table='IRIS_ID')\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812550_83527112","id":"20210302-002652_1368807412","dateCreated":"2021-01-13T01:09:25+0000","dateStarted":"2021-03-02T21:14:47+0000","dateFinished":"2021-03-02T21:14:48+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:95"},{"text":"%md\n\n.\nSplit the data set into train and test sets.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:48+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Split the data set into train and test sets.
.\n
Train an SVM model.
.\n
Create the MLX Global Feature Importance explainer gfi, using the f1_weighted metric.
.
\nInvoke gfi.explain to generate the global feature importance for the provided data. The explainer returns an ordered list of features along with their importance values. Features located at higher ranks have more impact on the model predictions. Importance values also represent relative feature importance. Here we see that petal length is the most important feature.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1632228361","id":"20210302-002652_1585407967","dateCreated":"2021-01-13T01:22:40+0000","dateStarted":"2021-03-02T21:14:55+0000","dateFinished":"2021-03-02T21:14:55+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:102"},{"text":"%python\n\nexplanation = gfi.explain(model, X, y,\ncase_id='CASE_ID', n_iter=10)\nexplanation\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:14:55+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Global Feature Importance: \n\t[0] petal length (cm): Value: 0.3496, Error: 0.0236\n\t[1] petal width (cm): Value: 0.2440, Error: 0.0332\n\t[2] sepal width (cm): Value: 0.0861, Error: 0.0197\n\t[3] sepal length (cm): Value: 0.0108, Error: 0.0124\n\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_-782370515","id":"20210302-002652_1988916332","dateCreated":"2021-01-13T01:22:58+0000","dateStarted":"2021-03-02T21:14:56+0000","dateFinished":"2021-03-02T21:15:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:103"},{"title":"Try it yourself","text":"%md\nBuild an in-database RandomForest model and compare the model's attribute importance ranking with that from *MLX*.","user":"OMLUSER02","dateUpdated":"2021-03-02T21:15:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","title":true,"editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Build an in-database RandomForest model and compare the model's attribute importance ranking with that from MLX.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812550_1752875015","id":"20210302-002652_543948810","dateCreated":"2021-01-20T05:10:19+0000","dateStarted":"2021-03-02T21:15:02+0000","dateFinished":"2021-03-02T21:15:02+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:104"},{"text":"%python\n\n# Insert code here\n\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:15:02+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812550_577511151","id":"20210302-002652_707777294","dateCreated":"2021-01-20T05:11:32+0000","dateStarted":"2021-03-02T21:15:02+0000","dateFinished":"2021-03-02T21:15:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:105"},{"text":"%md\n\n# End of Script\n","user":"OMLUSER02","dateUpdated":"2021-03-02T21:15:03+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nIn this lab, you will learn about the datastore feature of OML4Py.
\nDatastores exist in the user’s Oracle Database schema. A datastore, and the objects it contains, persist in the database until explicitly deleted. By using a datastore, you can store Python objects in a named datastore entry. This named datastore can then be used in subsequent Python sessions, and even be made available to other users or programs by granting/revoking read permissions.
\nPython objects, including OML DataFrame proxy objects, exist for the duration of the current Python session unless you explicitly save them. You can save one or more Python objects, including OML DataFrame proxy objects, to a named datastore and then load those objects in a later Python session. This is also useful when using embedded Python execution.
\nBy using a datastore, you can:
\nCopyright (c) 2021 Oracle Corporation
\nTo use OML4Py, first import the package oml. Import the Pandas package to work with oml DataFrames.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_1108743318","id":"20210224-211339_1894055839","dateCreated":"2021-01-10T19:49:44+0000","dateStarted":"2021-02-24T21:37:20+0000","dateFinished":"2021-02-24T21:37:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:23"},{"text":"%python\n\nimport pandas as pd\nimport oml\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219268_-1825825005","id":"20210224-211339_89120861","dateCreated":"2021-01-10T19:54:18+0000","dateStarted":"2021-02-24T21:37:21+0000","dateFinished":"2021-02-24T21:37:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:24"},{"title":"Step 2: Create Pandas DataFrames and load them into Autonomous Database","text":"%md\n\nIn this section, you will work with three data sets - *IRIS* data set, *Diabetes* data set, and *Boston* data set. \n\nYou will learn how to:\n\n* Get an OML DataFrame proxy object for the IRIS table and display the columns.\n* Create and display the columns for each data set\n* Use the dataframe to explore the datastore functionality\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:30+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"In this section, you will work with three data sets - IRIS data set, Diabetes data set, and Boston data set.
\nYou will learn how to:
\n.\n
Sync the IRIS table in the OMLUSER schema to an OML DataFrame proxy object.
.\n
Create the temporary diabetes table.
.\n
Create the temporary Boston table.
Save the IRIS data set and the temporary BOSTON proxy object to a datastore named ds_pydata, overwriting if the named datastore already exists. Note that you can store actual data objects in a datastore, but large data objects should remain as database tables for performance and scalability.
\nBy storing the BOSTON_TMP object, the temporary table will not be deleted when the session terminates.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_2141784995","id":"20210224-211339_1583489267","dateCreated":"2021-01-10T20:03:28+0000","dateStarted":"2021-02-24T21:37:35+0000","dateFinished":"2021-02-24T21:37:35+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:32"},{"text":"%md\n\n.\nSave the *IRIS* and *BOSTON* tables in the OML4Py Autonomous Database datastore.\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:35+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Save the IRIS and BOSTON tables in the OML4Py Autonomous Database datastore.
.\n
Add the temporary DIABETES_TMP proxy object to the same datastore. Note the use of append=True.
.\n
Save the IRIS table to a new datastore, and then list the datastores. Notice we see the datastore name, the number of objects in the datastore, the size in bytes consumed, when the datastore was create/updated, and any description provided by the user. Our two datastores ds_iris_data and ds_pydata are present, with the latter containing the three objects we added
This step illustrates how to store other types of objects in datastores. For this, you will create regression models using sklearn and OML4Py.
\nThe following script builds two regression models - regr1 and regr2. One model (regr1) uses the open source function LinearRegression() and regr2 uses the OML function oml.glm.
\nNote, this highlights that both open source models and in-database model proxy objects can be stored in a datastore. Like tables created using the oml.push function, default-named model proxy objects are dropped at the end of the database connection unless they are saved in a datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_-45603203","id":"20210224-211339_1430908498","dateCreated":"2021-01-10T21:18:16+0000","dateStarted":"2021-02-24T21:37:47+0000","dateFinished":"2021-02-24T21:37:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:39"},{"text":"%python\n\nfrom sklearn import linear_model\n\nregr1 = linear_model.LinearRegression()\nregr1.fit(boston.data, boston.target)\n\nregr2 = oml.glm(\"regression\")\nX = BOSTON_TMP.drop('Value')\ny = BOSTON_TMP['Value']\nregr2 = regr2.fit(X, y)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219268_1199894551","id":"20210224-211339_890485","dateCreated":"2021-01-10T21:19:50+0000","dateStarted":"2021-02-24T21:37:48+0000","dateFinished":"2021-02-24T21:37:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:40"},{"text":"%md\n\n.\nSave the objects *regr1* and *regr2* to the datastore named \"ds_pymodels\" and allow the read privilege to be granted to them.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:51+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Save the objects regr1 and regr2 to the datastore named “ds_pymodels” and allow the read privilege to be granted to them.
.\n
Grant the read privilege to all users by specifying user=None. Finally, list the datastores to which the read privilege has been granted.
Load all Python objects from a datastore to the global workspace and sort the result by name. Notice that they have the name specified in the dictionary when saved.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_2071884467","id":"20210224-211339_1544981158","dateCreated":"2021-01-10T21:22:09+0000","dateStarted":"2021-02-24T21:37:59+0000","dateFinished":"2021-02-24T21:37:59+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:45"},{"text":"%md\n.\nLoad the objects associated with the datastore named *ds_pydata* into memory\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:37:59+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Load the objects associated with the datastore named ds_pydata into memory
.\n
Load the Python object named regr2 from the datastore ds_pymodels to theglobal workspace. Display the model details for model regr2. With the boolean to_globals parameter, you can specify whether the objects are loaded to a global workspace or to a dictionary object. If the argument to_globals is True, then oml.ds.load function loads the objects into the global workspace. If the argument is False, then the function returns a dict object that contains pairs of object names and values.
Note, True is the default value for to_globals, so it it not necessary to specify this argument explicitly.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_2006554485","id":"20210224-211339_2145859244","dateCreated":"2021-01-10T21:23:40+0000","dateStarted":"2021-02-24T21:38:06+0000","dateFinished":"2021-02-24T21:38:06+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:48"},{"text":"%python\n\noml.ds.load(name=\"ds_pymodels\", objs=[\"regr2\"], to_globals=True)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:06+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"['regr2']\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_-173054131","id":"20210224-211339_390905932","dateCreated":"2021-01-10T21:24:22+0000","dateStarted":"2021-02-24T21:38:06+0000","dateFinished":"2021-02-24T21:38:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:49"},{"text":"%python\n\nregr2","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:12+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nAlgorithm Name: Generalized Linear Model\n\nMining Function: REGRESSION\n\nTarget: Value\n\nSettings: \n setting name setting value\n0 ALGO_NAME ALGO_GENERALIZED_LINEAR_MODEL\n1 GLMS_CONF_LEVEL .95\n2 GLMS_FTR_GENERATION GLMS_FTR_GENERATION_DISABLE\n3 GLMS_FTR_SELECTION GLMS_FTR_SELECTION_DISABLE\n4 ODMS_DETAILS ODMS_ENABLE\n5 ODMS_MISSING_VALUE_TREATMENT ODMS_MISSING_VALUE_AUTO\n6 ODMS_SAMPLING ODMS_SAMPLING_DISABLE\n7 PREP_AUTO ON\n\nComputed Settings: \n setting name setting value\n0 GLMS_CONV_TOLERANCE .0000050000000000000004\n1 GLMS_NUM_ITERATIONS 30\n2 GLMS_RIDGE_REGRESSION GLMS_RIDGE_REG_DISABLE\n3 GLMS_SOLVER GLMS_SOLVER_CHOL\n\nGlobal Statistics: \n attribute name attribute value\n0 ADJUSTED_R_SQUARE 0.73379\n1 AIC 1589.64\n2 COEFF_VAR 21.0595\n3 CONVERGED YES\n4 CORRECTED_TOTAL_DF 505\n5 CORRECTED_TOT_SS 42716.3\n6 DEPENDENT_MEAN 22.5328\n7 ERROR_DF 492\n8 ERROR_MEAN_SQUARE 22.5179\n9 ERROR_SUM_SQUARES 11078.8\n10 F_VALUE 108.077\n11 GMSEP 23.1597\n12 HOCKING_SP 0.045861\n13 J_P 23.1409\n14 MODEL_DF 13\n15 MODEL_F_P_VALUE 0\n16 MODEL_MEAN_SQUARE 2433.65\n17 MODEL_SUM_SQUARES 31637.5\n18 NUM_PARAMS 14\n19 NUM_ROWS 506\n20 RANK_DEFICIENCY 0\n21 ROOT_MEAN_SQ 4.7453\n22 R_SQ 0.740643\n23 SBIC 1648.81\n24 VALID_COVARIANCE_MATRIX YES\n\nAttributes: \nAGE\nB\nCHAS\nCRIM\nDIS\nINDUS\nLSTAT\nNOX\nPTRATIO\nRAD\nRM\nTAX\nZN\n\nPartition: NO\n\nCoefficients: \n\n name level estimate\n0 (Intercept) None 36.459488\n1 AGE None 0.000692\n2 B None 0.009312\n3 CHAS None 2.686734\n4 CRIM None -0.108011\n5 DIS None -1.475567\n6 INDUS None 0.020559\n7 LSTAT None -0.524758\n8 NOX None -17.766611\n9 PTRATIO None -0.952747\n10 RAD None 0.306049\n11 RM None 3.809865\n12 TAX None -0.012335\n13 ZN None 0.046420\n\nFit Details: \n\n name value\n0 ADJUSTED_R_SQUARE 0.733790\n1 AIC 1589.642798\n2 COEFF_VAR 21.059508\n3 CORRECTED_TOTAL_DF 505.000000\n4 CORRECTED_TOT_SS 42716.295415\n5 DEPENDENT_MEAN 22.532806\n6 ERROR_DF 492.000000\n7 ERROR_MEAN_SQUARE 22.517855\n8 ERROR_SUM_SQUARES 11078.784578\n9 F_VALUE 108.076666\n10 GMSEP 23.159731\n11 HOCKING_SP 0.045861\n12 J_P 23.140878\n13 MODEL_CONVERGED 1.000000\n14 MODEL_DF 13.000000\n15 MODEL_F_P_VALUE 0.000000\n16 MODEL_MEAN_SQUARE 2433.654680\n17 MODEL_SUM_SQUARES 31637.510837\n18 NUM_PARAMS 14.000000\n19 NUM_ROWS 506.000000\n20 RANK_DEFICIENCY 0.000000\n21 ROOT_MEAN_SQ 4.745298\n22 R_SQ 0.740643\n23 SBIC 1648.814312\n24 VALID_COVARIANCE_MATRIX 1.000000\n\nRank: \n\n14\n\nDeviance: \n\n11078.784578\n\nAIC: \n\n1590\n\nNull Deviance: \n\n42716.295415\n\nDF Residual: \n\n492.0\n\nDF Null: \n\n505.0\n\nConverged: \n\nTrue\n\n\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_-1888765752","id":"20210224-211339_1120145859","dateCreated":"2021-01-16T19:05:32+0000","dateStarted":"2021-02-24T21:38:12+0000","dateFinished":"2021-02-24T21:38:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:50"},{"text":"%md\n\n.\nRun the following command to load the named Python object (regression model) *regr1*, from the datastore to the user's workspace.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:12+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Run the following command to load the named Python object (regression model) regr1, from the datastore to the user's workspace.
List the datastores to which other users have been granted the read privilege. Notice that this is shared with PUBLIC, so other users have access to this datastore as well.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_1707702510","id":"20210224-211339_842861112","dateCreated":"2021-01-10T21:26:16+0000","dateStarted":"2021-02-24T21:38:17+0000","dateFinished":"2021-02-24T21:38:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:53"},{"text":"%python\n\noml.ds.dir(dstype=\"all\")[['owner', 'datastore_name', 'object_count']]\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" owner datastore_name object_count\n0 OMLUSER IRIS 1\n1 OMLUSER ds_pydata 3\n2 OMLUSER ds_pymodels 2\n3 OMLUSER15 ds_pymodels 2\n4 OMLUSER20 ds_pymodels 2\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_-1291411684","id":"20210224-211339_1343578037","dateCreated":"2021-01-10T21:42:48+0000","dateStarted":"2021-02-24T21:38:18+0000","dateFinished":"2021-02-24T21:38:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%md\n\n.\nList the datastores to which other users have been granted the read privilege.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the datastores to which other users have been granted the read privilege.
This step shows how to view and describe the content of a datastore. This example highlights the ds_pydata datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_999039049","id":"20210224-211339_1827904479","dateCreated":"2021-01-10T21:43:47+0000","dateStarted":"2021-02-24T21:38:24+0000","dateFinished":"2021-02-24T21:38:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:57"},{"text":"%md\n\n.\nList the content of datastore *ds_pydata*. The script returns the description of three proxy objects - *iris*, *oml_boston*, and *oml_diabetes*. It lists the class, \nsize, length, row and column count for each object. Notice that the two proxy objects and one Pandas DataFrame are listed.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:25+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the content of datastore ds_pydata. The script returns the description of three proxy objects - iris, oml_boston, and oml_diabetes. It lists the class,\n
size, length, row and column count for each object. Notice that the two proxy objects and one Pandas DataFrame are listed.
This section shows how to revoke read privilege, show datastores to which read privilege has been granted, and again grant read privilege to a user.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_1314716941","id":"20210224-211339_1529220730","dateCreated":"2021-01-10T21:48:14+0000","dateStarted":"2021-02-24T21:38:28+0000","dateFinished":"2021-02-24T21:38:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:60"},{"text":"%md\n\n.\nRevoke the read privilege from every user by specifying *user=None* on the datastore *ds_pymodels*. The result is empty. Notice that the type is *datastore*. In the next lab we'll see that this same function can be used for scripts in the script repository. \n","user":"OMLUSER","dateUpdated":"2021-02-24T21:38:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Revoke the read privilege from every user by specifying user=None on the datastore ds_pymodels. The result is empty. Notice that the type is datastore. In the next lab we'll see that this same function can be used for scripts in the script repository.
.\n
Grant the read privilege to PUBLIC by using user=None.\n
If you wanted to grant read privilege to a single user, you can do something like:
oml.grant(name=\"ds_pymodels\", typ=\"datastore\", user=\"OMLUSER\")\nThis step shows how to use the oml.ds.delete function to delete datastores or datastore contents.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_452566083","id":"20210224-211339_1940298635","dateCreated":"2021-01-10T21:58:40+0000","dateStarted":"2021-02-05T20:49:33+0000","dateFinished":"2021-02-05T20:49:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:65"},{"text":"%md\n\n.\nFirst, delete the contents of datastore *ds_pydata*, then delete the datastore itself. Next, use regular expression pattern matching to delete the datastore *ds_pymodels*.\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:49:33+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
First, delete the contents of datastore ds_pydata, then delete the datastore itself. Next, use regular expression pattern matching to delete the datastore ds_pymodels.
Delete the IRIS table from datastore IRIS, then delete the entire datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219268_1205023547","id":"20210224-211339_440371078","dateCreated":"2021-01-21T01:01:48+0000","dateStarted":"2021-02-05T20:49:39+0000","dateFinished":"2021-02-05T20:49:39+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:68"},{"text":"%python\n\n# Insert code here\n\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:49:39+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219268_1574340852","id":"20210224-211339_1568812425","dateCreated":"2021-01-21T01:02:59+0000","dateStarted":"2021-02-05T20:49:39+0000","dateFinished":"2021-02-05T20:49:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:69"},{"text":"%md\n\n# End of Script\n","user":"OMLUSER03","dateUpdated":"2021-02-05T20:49:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nIn this lab, you will learn about the datastore feature of OML4Py.
\nDatastores exist in the user’s Oracle Database schema. A datastore, and the objects it contains, persist in the database until explicitly deleted. By using a datastore, you can store Python objects in a named datastore entry. This named datastore can then be used in subsequent Python sessions, and even be made available to other users or programs by granting/revoking read permissions.
\nPython objects, including OML DataFrame proxy objects, exist for the duration of the current Python session unless you explicitly save them. You can save one or more Python objects, including OML DataFrame proxy objects, to a named datastore and then load those objects in a later Python session. This is also useful when using embedded Python execution.
\nBy using a datastore, you can:
\nCopyright (c) 2021 Oracle Corporation
\nTo use OML4Py, first import the package oml. Import the Pandas package to work with oml DataFrames.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-1470953987","id":"20210302-002652_1068144548","dateCreated":"2021-01-10T19:49:44+0000","dateStarted":"2021-03-02T19:16:41+0000","dateFinished":"2021-03-02T19:16:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%python\n\nimport pandas as pd\nimport oml\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:16:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812360_-1849453039","id":"20210302-002652_1201849809","dateCreated":"2021-01-10T19:54:18+0000","dateStarted":"2021-03-02T19:16:42+0000","dateFinished":"2021-03-02T19:16:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"title":"Step 2: Create Pandas DataFrames and load them into Autonomous Database","text":"%md\n\nIn this section, you will work with three data sets - *IRIS* data set, *Diabetes* data set, and *Boston* data set. \n\nYou will learn how to:\n\n* Get an OML DataFrame proxy object for the IRIS table and display the columns.\n* Create and display the columns for each data set\n* Use the dataframe to explore the datastore functionality\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:16:42+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"In this section, you will work with three data sets - IRIS data set, Diabetes data set, and Boston data set.
\nYou will learn how to:
\n.\n
Sync the IRIS table in the OMLUSER schema to an OML DataFrame proxy object.
.\n
Create the temporary diabetes table.
.\n
Create the temporary Boston table.
Save the IRIS data set and the temporary BOSTON proxy object to a datastore named ds_pydata, overwriting if the named datastore already exists. Note that you can store actual data objects in a datastore, but large data objects should remain as database tables for performance and scalability.
\nBy storing the BOSTON_TMP object, the temporary table will not be deleted when the session terminates.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_1866664803","id":"20210302-002652_2025442086","dateCreated":"2021-01-10T20:03:28+0000","dateStarted":"2021-03-02T19:16:47+0000","dateFinished":"2021-03-02T19:16:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:50"},{"text":"%md\n\n.\nSave the *IRIS* and *BOSTON* tables in the OML4Py Autonomous Database datastore.\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:16:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Save the IRIS and BOSTON tables in the OML4Py Autonomous Database datastore.
.\n
Add the temporary DIABETES_TMP proxy object to the same datastore. Note the use of append=True.
.\n
Save the IRIS table to a new datastore, and then list the datastores. Notice we see the datastore name, the number of objects in the datastore, the size in bytes consumed, when the datastore was create/updated, and any description provided by the user. Our two datastores ds_iris_data and ds_pydata are present, with the latter containing the three objects we added
This step illustrates how to store other types of objects in datastores. For this, you will create regression models using sklearn and OML4Py.
\nThe following script builds two regression models - regr1 and regr2. One model (regr1) uses the open source function LinearRegression() and regr2 uses the OML function oml.glm.
\nNote, this highlights that both open source models and in-database model proxy objects can be stored in a datastore. Like tables created using the oml.push function, default-named model proxy objects are dropped at the end of the database connection unless they are saved in a datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-1548115155","id":"20210302-002652_1700326650","dateCreated":"2021-01-10T21:18:16+0000","dateStarted":"2021-03-02T19:17:02+0000","dateFinished":"2021-03-02T19:17:02+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:57"},{"text":"%python\n\nfrom sklearn import linear_model\n\nregr1 = linear_model.LinearRegression()\nregr1.fit(boston.data, boston.target)\n\nregr2 = oml.glm(\"regression\")\nX = BOSTON_TMP.drop('Value')\ny = BOSTON_TMP['Value']\nregr2 = regr2.fit(X, y)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:02+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812360_432464009","id":"20210302-002652_1906366990","dateCreated":"2021-01-10T21:19:50+0000","dateStarted":"2021-03-02T19:17:03+0000","dateFinished":"2021-03-02T19:17:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:58"},{"text":"%md\n\n.\nSave the objects *regr1* and *regr2* to the datastore named \"ds_pymodels\" and allow the read privilege to be granted to them.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:11+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Save the objects regr1 and regr2 to the datastore named “ds_pymodels” and allow the read privilege to be granted to them.
.\n
Grant the read privilege to all users by specifying user=None. Finally, list the datastores to which the read privilege has been granted.
Load all Python objects from a datastore to the global workspace and sort the result by name. Notice that they have the name specified in the dictionary when saved.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_448633542","id":"20210302-002652_1108140185","dateCreated":"2021-01-10T21:22:09+0000","dateStarted":"2021-03-02T19:17:19+0000","dateFinished":"2021-03-02T19:17:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:63"},{"text":"%md\n.\nLoad the objects associated with the datastore named *ds_pydata* into memory\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Load the objects associated with the datastore named ds_pydata into memory
.\n
Load the Python object named regr2 from the datastore ds_pymodels to theglobal workspace. Display the model details for model regr2. With the boolean to_globals parameter, you can specify whether the objects are loaded to a global workspace or to a dictionary object. If the argument to_globals is True, then oml.ds.load function loads the objects into the global workspace. If the argument is False, then the function returns a dict object that contains pairs of object names and values.
Note, True is the default value for to_globals, so it it not necessary to specify this argument explicitly.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_818695247","id":"20210302-002652_1746610666","dateCreated":"2021-01-10T21:23:40+0000","dateStarted":"2021-03-02T19:17:26+0000","dateFinished":"2021-03-02T19:17:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:66"},{"text":"%python\n\noml.ds.load(name=\"ds_pymodels\", objs=[\"regr2\"], to_globals=True)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"['regr2']\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-2089291841","id":"20210302-002652_1295389784","dateCreated":"2021-01-10T21:24:22+0000","dateStarted":"2021-03-02T19:17:27+0000","dateFinished":"2021-03-02T19:17:32+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:67"},{"text":"%python\n\nregr2","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:32+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"\nAlgorithm Name: Generalized Linear Model\n\nMining Function: REGRESSION\n\nTarget: Value\n\nSettings: \n setting name setting value\n0 ALGO_NAME ALGO_GENERALIZED_LINEAR_MODEL\n1 GLMS_CONF_LEVEL .95\n2 GLMS_FTR_GENERATION GLMS_FTR_GENERATION_DISABLE\n3 GLMS_FTR_SELECTION GLMS_FTR_SELECTION_DISABLE\n4 ODMS_DETAILS ODMS_ENABLE\n5 ODMS_MISSING_VALUE_TREATMENT ODMS_MISSING_VALUE_AUTO\n6 ODMS_SAMPLING ODMS_SAMPLING_DISABLE\n7 PREP_AUTO ON\n\nComputed Settings: \n setting name setting value\n0 GLMS_CONV_TOLERANCE .0000050000000000000004\n1 GLMS_NUM_ITERATIONS 30\n2 GLMS_RIDGE_REGRESSION GLMS_RIDGE_REG_DISABLE\n3 GLMS_SOLVER GLMS_SOLVER_CHOL\n\nGlobal Statistics: \n attribute name attribute value\n0 ADJUSTED_R_SQUARE 0.73379\n1 AIC 1589.64\n2 COEFF_VAR 21.0595\n3 CONVERGED YES\n4 CORRECTED_TOTAL_DF 505\n5 CORRECTED_TOT_SS 42716.3\n6 DEPENDENT_MEAN 22.5328\n7 ERROR_DF 492\n8 ERROR_MEAN_SQUARE 22.5179\n9 ERROR_SUM_SQUARES 11078.8\n10 F_VALUE 108.077\n11 GMSEP 23.1597\n12 HOCKING_SP 0.045861\n13 J_P 23.1409\n14 MODEL_DF 13\n15 MODEL_F_P_VALUE 0\n16 MODEL_MEAN_SQUARE 2433.65\n17 MODEL_SUM_SQUARES 31637.5\n18 NUM_PARAMS 14\n19 NUM_ROWS 506\n20 RANK_DEFICIENCY 0\n21 ROOT_MEAN_SQ 4.7453\n22 R_SQ 0.740643\n23 SBIC 1648.81\n24 VALID_COVARIANCE_MATRIX YES\n\nAttributes: \nAGE\nB\nCHAS\nCRIM\nDIS\nINDUS\nLSTAT\nNOX\nPTRATIO\nRAD\nRM\nTAX\nZN\n\nPartition: NO\n\nCoefficients: \n\n name level estimate\n0 (Intercept) None 36.459488\n1 AGE None 0.000692\n2 B None 0.009312\n3 CHAS None 2.686734\n4 CRIM None -0.108011\n5 DIS None -1.475567\n6 INDUS None 0.020559\n7 LSTAT None -0.524758\n8 NOX None -17.766611\n9 PTRATIO None -0.952747\n10 RAD None 0.306049\n11 RM None 3.809865\n12 TAX None -0.012335\n13 ZN None 0.046420\n\nFit Details: \n\n name value\n0 ADJUSTED_R_SQUARE 0.733790\n1 AIC 1589.642798\n2 COEFF_VAR 21.059508\n3 CORRECTED_TOTAL_DF 505.000000\n4 CORRECTED_TOT_SS 42716.295415\n5 DEPENDENT_MEAN 22.532806\n6 ERROR_DF 492.000000\n7 ERROR_MEAN_SQUARE 22.517855\n8 ERROR_SUM_SQUARES 11078.784578\n9 F_VALUE 108.076666\n10 GMSEP 23.159731\n11 HOCKING_SP 0.045861\n12 J_P 23.140878\n13 MODEL_CONVERGED 1.000000\n14 MODEL_DF 13.000000\n15 MODEL_F_P_VALUE 0.000000\n16 MODEL_MEAN_SQUARE 2433.654680\n17 MODEL_SUM_SQUARES 31637.510837\n18 NUM_PARAMS 14.000000\n19 NUM_ROWS 506.000000\n20 RANK_DEFICIENCY 0.000000\n21 ROOT_MEAN_SQ 4.745298\n22 R_SQ 0.740643\n23 SBIC 1648.814312\n24 VALID_COVARIANCE_MATRIX 1.000000\n\nRank: \n\n14\n\nDeviance: \n\n11078.784578\n\nAIC: \n\n1590\n\nNull Deviance: \n\n42716.295415\n\nDF Residual: \n\n492.0\n\nDF Null: \n\n505.0\n\nConverged: \n\nTrue\n\n\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-1897570242","id":"20210302-002652_670378094","dateCreated":"2021-01-16T19:05:32+0000","dateStarted":"2021-03-02T19:17:32+0000","dateFinished":"2021-03-02T19:17:33+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:68"},{"text":"%md\n\n.\nRun the following command to load the named Python object (regression model) *regr1*, from the datastore to the user's workspace.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:33+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Run the following command to load the named Python object (regression model) regr1, from the datastore to the user's workspace.
List the datastores to which other users have been granted the read privilege. Notice that this is shared with PUBLIC, so other users have access to this datastore as well.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_932886228","id":"20210302-002652_402246046","dateCreated":"2021-01-10T21:26:16+0000","dateStarted":"2021-03-02T19:17:38+0000","dateFinished":"2021-03-02T19:17:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:71"},{"text":"%python\n\noml.ds.dir(dstype=\"all\")[['owner', 'datastore_name', 'object_count']]\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:38+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" owner datastore_name object_count\n0 OMLUSER02 IRIS 1\n1 OMLUSER02 ds-1 3\n2 OMLUSER02 ds_pydata 3\n3 OMLUSER02 ds_pymodels 2\n4 OMLUSER15 ds_pymodels 2\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-636130828","id":"20210302-002652_423703895","dateCreated":"2021-01-10T21:42:48+0000","dateStarted":"2021-03-02T19:17:38+0000","dateFinished":"2021-03-02T19:17:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:72"},{"text":"%md\n\n.\nList the datastores to which other users have been granted the read privilege.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the datastores to which other users have been granted the read privilege.
This step shows how to view and describe the content of a datastore. This example highlights the ds_pydata datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-184201028","id":"20210302-002652_1483189024","dateCreated":"2021-01-10T21:43:47+0000","dateStarted":"2021-03-02T19:17:44+0000","dateFinished":"2021-03-02T19:17:44+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:75"},{"text":"%md\n\n.\nList the content of datastore *ds_pydata*. The script returns the description of three proxy objects - *iris*, *oml_boston*, and *oml_diabetes*. It lists the class, \nsize, length, row and column count for each object. Notice that the two proxy objects and one Pandas DataFrame are listed.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:44+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
List the content of datastore ds_pydata. The script returns the description of three proxy objects - iris, oml_boston, and oml_diabetes. It lists the class,\n
size, length, row and column count for each object. Notice that the two proxy objects and one Pandas DataFrame are listed.
This section shows how to revoke read privilege, show datastores to which read privilege has been granted, and again grant read privilege to a user.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_199041878","id":"20210302-002652_1133909759","dateCreated":"2021-01-10T21:48:14+0000","dateStarted":"2021-03-02T19:17:48+0000","dateFinished":"2021-03-02T19:17:48+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:78"},{"text":"%md\n\n.\nRevoke the read privilege from every user by specifying *user=None* on the datastore *ds_pymodels*. The result is empty. Notice that the type is *datastore*. In the next lab we'll see that this same function can be used for scripts in the script repository. \n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:48+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Revoke the read privilege from every user by specifying user=None on the datastore ds_pymodels. The result is empty. Notice that the type is datastore. In the next lab we'll see that this same function can be used for scripts in the script repository.
.\n
Grant the read privilege to OMLUSER.
This step shows how to use the oml.ds.delete function to delete datastores or datastore contents.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-1811829762","id":"20210302-002652_1866878349","dateCreated":"2021-01-10T21:58:40+0000","dateStarted":"2021-03-02T19:17:51+0000","dateFinished":"2021-03-02T19:17:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:83"},{"text":"%md\n\n.\nFirst, delete the contents of datastore *ds_pydata*, then delete the datastore itself. Next, use regular expression pattern matching to delete the datastore *ds_pymodels*.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:17:51+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
First, delete the contents of datastore ds_pydata, then delete the datastore itself. Next, use regular expression pattern matching to delete the datastore ds_pymodels.
Delete the IRIS table from datastore IRIS, then delete the entire datastore.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812360_-531069746","id":"20210302-002652_1087359159","dateCreated":"2021-01-21T01:01:48+0000","dateStarted":"2021-03-02T19:18:01+0000","dateFinished":"2021-03-02T19:18:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:86"},{"text":"%python\n\n# Insert code here\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:18:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812360_1825073012","id":"20210302-002652_498650886","dateCreated":"2021-01-21T01:02:59+0000","dateStarted":"2021-03-02T19:18:01+0000","dateFinished":"2021-03-02T19:18:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:87"},{"text":"%md\n\n# End of Script\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:18:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nIn this notebook, we highlight using OML4Py embedded Python execution and the Python script repository.
\nOML4Py stores named user-defined functions called scripts in the script repository. With the script repository, users can:
\nThe OML4Py script repository functions are:
\nEmbedded Python execution enables you to run user-defined Python functions in Python engines spawned in the Autonomous Database environment. These engines run alongside an OML Notebooks Python interpreter session. On Autonomous Database, OML4Py provides both a Python interface and a REST API interface for embedded Python execution. Use the OML4Py embedded Python execution REST API to query and invoke user-defined Python functions saved in the datastore. The result of the REST execution is a JSON representation of the value returned from the Python script.
\nThe OML4Py embedded Python execution functions are:
\nDownload the REST API notebook from Github to view examples here:
\nCopyright (c) 2021 Oracle Corporation
\nImport the oml library.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220119_1789229012","id":"20210224-211340_1175270679","dateCreated":"2021-01-11T17:53:16+0000","dateStarted":"2021-02-24T21:52:41+0000","dateFinished":"2021-02-24T21:52:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:23"},{"text":"%python\n\nimport oml\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:52:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201220120_-575338175","id":"20210224-211340_1595400719","dateCreated":"2021-01-11T19:05:58+0000","dateStarted":"2021-02-24T21:52:42+0000","dateFinished":"2021-02-24T21:52:50+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:24"},{"text":"%md\n.\nCreate the *iris* data.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:52:50+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create the iris data.
When working with embedded Python execution, a typical workflow is:
\nIn this section, you will validate your Python script, validate the user-defined function before invoking the embedded Python APIs. You will also build a linear regression model LinearRegression.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1558926798","id":"20210224-211340_1467151716","dateCreated":"2021-01-11T19:10:25+0000","dateStarted":"2021-02-24T21:52:53+0000","dateFinished":"2021-02-24T21:52:53+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:27"},{"text":"%md\n.\nRun the following script to create a linear model with sklearn.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:52:53+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Run the following script to create a linear model with sklearn.
.\n
Predict the petal width using the predict function.
.\n
Assess model quality using mean squared error and R^2.
.\n
Generate a scatterplot of the data along with a plot of the regression line.
In this step, you will build the same linear model, but using the embedded Python execution Python engines under control of the Autonomous Database environment. You will perform the following tasks:
\nNote, in the Python API, the embedded Python execution functions accept a function object or named string as input, and can return images as well as structured content.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1979551073","id":"20210224-211340_1561182138","dateCreated":"2021-01-11T19:22:58+0000","dateStarted":"2021-02-24T21:53:00+0000","dateFinished":"2021-02-24T21:53:00+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:36"},{"text":"%md\n.\nDefine the user-defined function *build_lm_1*.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:00+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the user-defined function build_lm_1.
.\n
Invoke your user-defined function to build the model and plot the petal length predictions.
oml.table_apply takes the OML DataFrame proxy object IRIS as input data and loads that data to the user-defined function as a pandas DataFrame. In this example, the user-defined function is passed as a Python string. You see that the model comes back as an OML object, which you can pull to the client to view the the linear model.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1605768357","id":"20210224-211340_82965292","dateCreated":"2021-01-11T19:31:10+0000","dateStarted":"2021-02-24T21:53:04+0000","dateFinished":"2021-02-24T21:53:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%python\n\nimport inspect\n\nbuild_lm_1 = \"\"\"def build_lm_1(dat):\n import oml\n from sklearn import linear_model\n import matplotlib.pyplot as plt\n\n lm = linear_model.LinearRegression()\n X = dat[[\"PETAL_WIDTH\"]]\n y = dat[[\"PETAL_LENGTH\"]]\n mod = lm.fit(X, y)\n\n pred = mod.predict(dat[[\"PETAL_WIDTH\"]])\n plt.scatter(dat.loc[:,\"PETAL_WIDTH\"], dat.loc[:,\"PETAL_LENGTH\"])\n plt.plot(dat[[\"PETAL_WIDTH\"]], pred, color='blue', linewidth=3)\n plt.grid(True)\n plt.title('Prediction of Petal Length')\n\n plt.xlabel('Petal Width')\n plt.ylabel('Petal Length')\n plt.show()\n return mod\"\"\"","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:05+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201220120_-2074547962","id":"20210224-211340_138541253","dateCreated":"2021-01-11T19:40:26+0000","dateStarted":"2021-02-24T21:53:05+0000","dateFinished":"2021-02-24T21:53:05+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"text":"%md\n.\nBy invoking **table_apply**, a Python engine is spawned and the user-defined function *build_lm_1* is invoked on that engine with the data referenced by *IRIS* being passed in as a pandas DataFrame. Part of the return value is the image, which is automatically displayed. In this example, we are passing the function object to the ***table_apply*** function.\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:05+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
By invoking table_apply, a Python engine is spawned and the user-defined function build_lm_1 is invoked on that engine with the data referenced by IRIS being passed in as a pandas DataFrame. Part of the return value is the image, which is automatically displayed. In this example, we are passing the function object to the table_apply function.
Use the embedded Python execution function oml.row_apply to run a user-defined function on chunks of rows, which is useful to perform scoring in parallel for native Python models. In this example, we pass the function to row_apply as a Python function object.
\nDefine the user-defined function score_lm_1 to make predictions (score data) using the data set and model passed in as arguments. It returns the predictions as a DataFrame object.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_1053858263","id":"20210224-211340_817018288","dateCreated":"2021-01-11T19:46:53+0000","dateStarted":"2021-02-24T21:53:09+0000","dateFinished":"2021-02-24T21:53:09+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:46"},{"text":"%python\n\ndef score_lm_1(dat, model):\n import pandas as pd\n from sklearn import linear_model\n pred = model.predict(dat[[\"PETAL_WIDTH\"]])\n return pd.concat([dat[['SPECIES', 'PETAL_LENGTH']], \n pd.DataFrame(pred, columns=['PRED_PETAL_LENGTH'])], axis=1)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:09+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201220120_1946860142","id":"20210224-211340_182849146","dateCreated":"2021-01-11T19:47:49+0000","dateStarted":"2021-02-24T21:53:10+0000","dateFinished":"2021-02-24T21:53:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:47"},{"text":"%md\n.\nUse ***row_apply*** to invoke this user-defined function and return a single OML DataFrame proxy object as the result. The ***row_apply*** function takes as arguments the proxy object *IRIS*, that we want 10 rows scored at a time (resulting in 15 function invocations), the user-defined function, the linear model object, and that we want the result to be returned as a single table by specifying the table definition.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:10+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use row_apply to invoke this user-defined function and return a single OML DataFrame proxy object as the result. The row_apply function takes as arguments the proxy object IRIS, that we want 10 rows scored at a time (resulting in 15 function invocations), the user-defined function, the linear model object, and that we want the result to be returned as a single table by specifying the table definition.
This step shows how to use the oml.group_apply function for model building. The group_apply apply function passes the OML DataFrame specified by the data argument to the user-defined func function as its first argument. The index argument specifies the columns of oml.DataFrame by which the database groups the data for processing by the user-defined Python function. The group_apply function can use data-parallel execution, in which one or more Python engines perform the same Python function on different groups of data.
\nIn this step, you build three models, one specific to each species and return them as a dictionary. These models are indexed by their corresponding index value.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_142337800","id":"20210224-211340_994735956","dateCreated":"2021-01-11T19:51:46+0000","dateStarted":"2021-02-24T21:53:13+0000","dateFinished":"2021-02-24T21:53:13+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:50"},{"text":"%python\n\ndef build_lm_g(dat):\n import oml\n from sklearn import linear_model\n lm = linear_model.LinearRegression()\n X = dat[[\"PETAL_WIDTH\"]]\n y = dat[[\"PETAL_LENGTH\"]]\n mod = lm.fit(X, y)\n return mod\n\nmod = oml.group_apply(IRIS[:,[\"PETAL_LENGTH\",\"PETAL_WIDTH\",\"SPECIES\"]], \n index=oml.DataFrame(IRIS['SPECIES']), \n func=build_lm_g, \n parallel=2)\n \nprint(\"Type:\",type(mod))\nmod\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:13+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Type:.\n
Modify the user-defined function to save the resulting model in the same named datastore. A datastore allows storing Python objects in the database under a provided name. The stored Python object assumes the variable name assigned in the Python environment. Here, we construct a name concatenating mod_ as a prefix and the corresponding Species value.
.\n
Use group_apply to invoke the user-defined function and list the resulting models, which are a dictionary of three elements each assigned the model object name. The group_apply function takes the data, the index parameter that specifies the column or columns to partition on, the user-defined function, and that we wish to automatically connect to the database from the Python engine. Connecting to the database is necessary when using the datastore functionality.
Note, if the datastore exists, then delete it so that the group_apply function completes successfully.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1454615086","id":"20210224-211340_1320265617","dateCreated":"2021-01-11T19:59:30+0000","dateStarted":"2021-02-24T21:53:16+0000","dateFinished":"2021-02-24T21:53:16+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%python\n\ntry:\n oml.ds.delete('ds-1')\nexcept:\n print(\"Datastore not found\")\n\nres = oml.group_apply(IRIS[:,[\"PETAL_LENGTH\",\"PETAL_WIDTH\",\"SPECIES\"]], \n index=oml.DataFrame(IRIS['SPECIES']), \n func=build_lm_2, dsname=\"ds-1\")\n\nprint(\"Outcome:\",res)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:17+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Datastore not found\nOutcome: {'virginica': 'mod_virginica', 'setosa': 'mod_setosa', 'versicolor': 'mod_versicolor'}\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-319938624","id":"20210224-211340_1721787437","dateCreated":"2021-01-11T20:00:33+0000","dateStarted":"2021-02-24T21:53:17+0000","dateFinished":"2021-02-24T21:53:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:55"},{"text":"%md\n.\nHere, the model object names are *mod_versicolor*, *mod_virginica*, and *mod_setosa*. When you load the datastore, you get the three models loaded into the client Python engine, assigned to their respective variables.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Here, the model object names are mod_versicolor, mod_virginica, and mod_setosa. When you load the datastore, you get the three models loaded into the client Python engine, assigned to their respective variables.
Use the group_apply function to count the number of each species in the data set.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_1167221733","id":"20210224-211340_525248762","dateCreated":"2021-01-11T20:06:56+0000","dateStarted":"2021-02-24T21:53:21+0000","dateFinished":"2021-02-24T21:53:21+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:57"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:21+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201220120_-1676074025","id":"20210224-211340_1543510445","dateCreated":"2021-01-11T20:07:51+0000","dateStarted":"2021-02-24T21:53:22+0000","dateFinished":"2021-02-24T21:53:22+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:58"},{"title":"Step 5: Invoke a function N times","text":"%md\n\nThe ***index_apply*** function allows the same function to be invoked a specified number of times. The first argument to the user-defined function is an index number for the function execution. For example, if the ***times*** argument is 10, each function invocation will receive a distinct value between 1 and 10. This can be used, e.g., for selecting behavior within the function or setting a random seed for Monte Carlo analysis.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:22+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"The index_apply function allows the same function to be invoked a specified number of times. The first argument to the user-defined function is an index number for the function execution. For example, if the times argument is 10, each function invocation will receive a distinct value between 1 and 10. This can be used, e.g., for selecting behavior within the function or setting a random seed for Monte Carlo analysis.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_695207631","id":"20210224-211340_971284236","dateCreated":"2021-01-11T20:05:15+0000","dateStarted":"2021-02-24T21:53:23+0000","dateFinished":"2021-02-24T21:53:23+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:59"},{"text":"%python\n\ndef compute_random_mean(index):\n import numpy as np\n import scipy\n from statistics import mean \n np.random.seed(index)\n res = np.random.random((100,1))*10\n return mean(res[1])\n \nprint(\"Test: \", compute_random_mean(3))\n\nres = oml.index_apply(times=12, func=compute_random_mean)\nres\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:23+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Test: 7.081478226181048\n[7.203244934421581, 0.25926231827891333, 7.081478226181048, 5.4723224917572235, 8.707323061773764, 3.3197980530117723, 7.7991879224011464, 9.68540662820932, 5.018745921487388, 0.207519493594015, 0.19475241487624584, 7.400496965154048]\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1859571265","id":"20210224-211340_273398689","dateCreated":"2021-01-11T20:06:34+0000","dateStarted":"2021-02-24T21:53:23+0000","dateFinished":"2021-02-24T21:53:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:60"},{"title":"Step 6: Return multiple images from Embedded Python Execution","text":"%md\n\nThis step shows how to create a function *RandomRedDots* that creates and returns a simple DataFrame and generates two scatterplots of random red dots. First, create *RandomRedDots* and save it in the Python Script Repository, and then define and run the native Python function.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:24+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"This step shows how to create a function RandomRedDots that creates and returns a simple DataFrame and generates two scatterplots of random red dots. First, create RandomRedDots and save it in the Python Script Repository, and then define and run the native Python function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-236484182","id":"20210224-211340_2057885738","dateCreated":"2021-01-11T20:11:27+0000","dateStarted":"2021-02-24T21:53:24+0000","dateFinished":"2021-02-24T21:53:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:61"},{"text":"%python\n\nRandomRedDots = \"\"\"def RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df\"\"\"\n\n\noml.script.create(\"RandomRedDots\", func=RandomRedDots, is_global=True, overwrite=True)\n\nprint(RandomRedDots)\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:24+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"def RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-463658455","id":"20210224-211340_641297315","dateCreated":"2021-01-11T20:38:19+0000","dateStarted":"2021-02-24T21:53:25+0000","dateFinished":"2021-02-24T21:53:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:62"},{"text":"%python\n\ndef RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:25+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201220120_-1487351964","id":"20210224-211340_993561751","dateCreated":"2021-01-18T04:06:24+0000","dateStarted":"2021-02-24T21:53:26+0000","dateFinished":"2021-02-24T21:53:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:63"},{"text":"%python\n\nRandomRedDots()","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"
.\n
Use the oml.do_eval function to invoke the function RandomRedDots. The do_eval function does not automatically pass data to the function, it merely invokes it. Here, we show accepting the default argument values for RandomRedDots.
.\n
In this next example, we modify the function to use subplots, thereby creating separate figure objects for the scatter plots. Store this in the script repository as RandomRedDots2 and invoke the function to see the results. As expected, you get both plots.
Note, when you call RandomRedDots2 using embedded Python execution, you will get both plots as shown in the result.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_889702079","id":"20210224-211340_1478243150","dateCreated":"2021-01-18T04:05:28+0000","dateStarted":"2021-02-24T21:53:30+0000","dateFinished":"2021-02-24T21:53:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:67"},{"text":"%md\n.\nDefine the *RandomRedDots2* function that generates two scatter plots, and returns a 2 column DataFrame. Note that we can pass arguments to these functions, here, *num_dots_1* and *num_dots_2*.","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:31+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the RandomRedDots2 function that generates two scatter plots, and returns a 2 column DataFrame. Note that we can pass arguments to these functions, here, num_dots_1 and num_dots_2.
.\n
Use the oml.do_eval function to call the function RandomRedDots2. In this invocation, we're specifying arguments to do_eval for num_dots_1 and num_dots_2. These are specified as you would any other argument to do_eval. This applies to the other embedded Python functions as well.
OML4Py stores named user-defined functions called scripts in the script repository. You can make scripts either private or global. A private script is available only to the owner. A global script is available to any user. For private scripts, the owner of the script may grant the read privilege to other users or revoke that privilege.
\nTo illustrate using the Python Script Repository, you will define a function build_lm_str that will fit a regression model. Using this function, you will then create a script named MyLM_function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_1902009689","id":"20210224-211340_14218496","dateCreated":"2021-01-18T01:54:21+0000","dateStarted":"2021-02-24T21:53:34+0000","dateFinished":"2021-02-24T21:53:34+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:72"},{"text":"%md\n.\nTo store a user-defined function in the script repository, it must be presented as a named string. Define the function as a string, *build_lm_str*. Note the use of triple quotes to enable formatting. ","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:34+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
To store a user-defined function in the script repository, it must be presented as a named string. Define the function as a string, build_lm_str. Note the use of triple quotes to enable formatting.
.\n
View the string that you just created.
Use the function oml.script.create to create a script MyLM_function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1774042471","id":"20210224-211340_565824715","dateCreated":"2021-01-18T02:10:27+0000","dateStarted":"2021-02-24T21:53:38+0000","dateFinished":"2021-02-24T21:53:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:77"},{"text":"%md\n.\nYou can use the ***oml.script.drop*** function to drop a named script, here shown in a try-except block. Then, use the ***oml.script.create*** function to create the script *MyLM_function*. Note that we have the *overwrite* argument set to *True* in case the function already exists.","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:38+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
You can use the oml.script.drop function to drop a named script, here shown in a try-except block. Then, use the oml.script.create function to create the script MyLM_function. Note that we have the overwrite argument set to True in case the function already exists.
List the scripts that are available only to the current user with oml.script.dir with sctype set to user.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_1209338535","id":"20210224-211340_253496692","dateCreated":"2021-01-18T02:14:33+0000","dateStarted":"2021-02-24T21:53:40+0000","dateFinished":"2021-02-24T21:53:40+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:81"},{"text":"%python\n\noml.script.dir(sctype='user')\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:40+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" name ... date\n0 MyLM_function ... 2021-02-24 21:53:40\n\n[1 rows x 4 columns]\n"}]},"interrupted":false,"jobName":"paragraph_1614201220120_-1857865118","id":"20210224-211340_1331265744","dateCreated":"2021-01-18T02:15:16+0000","dateStarted":"2021-02-24T21:53:41+0000","dateFinished":"2021-02-24T21:53:41+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:82"},{"text":"%md\n.\nGrant the read privilege on the *MyLM_function* script to PUBLIC with ***oml.grant***. \nNotice that the type is specified as *pyqscript*. This same function is used to grant privileges on datastores. \n\nIf you wanted to grant read privilege to a specific user, you would use something like:\n oml.grant(name=\"MyLM_function\", typ=\"pyqscript\", user=\"OMLUSER\")\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:41+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Grant the read privilege on the MyLM_function script to PUBLIC with oml.grant.\n
Notice that the type is specified as pyqscript. This same function is used to grant privileges on datastores.
If you wanted to grant read privilege to a specific user, you would use something like:
\noml.grant(name=\"MyLM_function\", typ=\"pyqscript\", user=\"OMLUSER\")\n.\n
List the scripts to which the read privilege has been granted using oml.script.dir and sctype set to grant.
.\n
Load the named function into the Python engine for use as a typical Python function using oml.script.load.
.\n
Extract the function text string from the function object and use this to save in the script repository using oml.script_create.
Use the function oml.script.dir to list all the available scripts.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220125_-639598540","id":"20210224-211340_2001720166","dateCreated":"2021-01-18T02:19:45+0000","dateStarted":"2021-02-24T21:53:48+0000","dateFinished":"2021-02-24T21:53:48+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:92"},{"text":"%python\n\noml.script.dir(sctype=\"all\")\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:48+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" owner name ... description date\n0 OMLUSER MyLM_function ... None 2021-02-24 21:53:40\n1 OMLUSER MyTEST_function ... None 2021-02-24 21:53:47\n2 PYQSYS RandomRedDots ... None 2021-02-24 21:53:25\n3 PYQSYS RandomRedDots2 ... None 2021-02-24 21:53:32\n\n[4 rows x 5 columns]\n"}]},"interrupted":false,"jobName":"paragraph_1614201220125_-1499548891","id":"20210224-211340_417184380","dateCreated":"2021-01-18T02:20:01+0000","dateStarted":"2021-02-24T21:53:49+0000","dateFinished":"2021-02-24T21:53:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:93"},{"text":"%md\n.\nInvoke ***table_apply*** on *build_lm_str* and *loaded_str*. Recall that these strings represent the same function - *build_lm_str* was saved to the script repository after assigning the function to a string object. *loaded_str* is the string representation of the function extracted using ***get_source().read()***.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:49+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Invoke table_apply on build_lm_str and loaded_str. Recall that these strings represent the same function - build_lm_str was saved to the script repository after assigning the function to a string object. loaded_str is the string representation of the function extracted using get_source().read().
A global function is one that can be accessed by any user.
\nIn this step, you will define and save a global function build_lm3. You will then call function build_lm3 to build another model.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201220125_-1624606786","id":"20210224-211340_1015069729","dateCreated":"2021-01-18T02:20:18+0000","dateStarted":"2021-02-24T21:53:52+0000","dateFinished":"2021-02-24T21:53:52+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:97"},{"text":"%md\n.\nDefine the function *build_lm3* as a named string and save it to the script repository.","user":"OMLUSER","dateUpdated":"2021-02-24T21:53:52+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the function build_lm3 as a named string and save it to the script repository.
.\n
Use the oml.table_apply function to invoke this function with the IRIS table. The oml.table_apply function spawns a Python engine, loads the specified user-defined function, and passes the data referenced by the IRIS proxy object as a pandas DataFrame in the first argument of the user-defined function. The resulting scikit-learn linear model is returned in res.
In this step, you will perform the following:
\n.\n
First drop the private script MyLM_function, then drop the global script MyGlobalML_function, and finally list the available scripts.
Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nIn this notebook, we highlight using OML4Py embedded Python execution and the Python script repository.
\nOML4Py stores named user-defined functions called scripts in the script repository. With the script repository, users can:
\nThe OML4Py script repository functions are:
\nEmbedded Python execution enables you to run user-defined Python functions in Python engines spawned in the Autonomous Database environment. These engines run alongside an OML Notebooks Python interpreter session. On Autonomous Database, OML4Py provides both a Python interface and a REST API interface for embedded Python execution. Use the OML4Py embedded Python execution REST API to query and invoke user-defined Python functions saved in the datastore. The result of the REST execution is a JSON representation of the value returned from the Python script.
\nThe OML4Py embedded Python execution functions are:
\nDownload the REST API notebook from Github to view examples here:
\nCopyright (c) 2021 Oracle Corporation
\nImport the oml library.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_1951267739","id":"20210302-002652_1281186904","dateCreated":"2021-01-11T17:53:16+0000","dateStarted":"2021-03-02T19:23:01+0000","dateFinished":"2021-03-02T19:23:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:41"},{"text":"%python\n\nimport oml\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812115_1667702196","id":"20210302-002652_1686303762","dateCreated":"2021-01-11T19:05:58+0000","dateStarted":"2021-03-02T19:23:01+0000","dateFinished":"2021-03-02T19:23:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:42"},{"text":"%md\n.\nCreate the *iris* data.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:10+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Create the iris data.
When working with embedded Python execution, a typical workflow is:
\nIn this section, you will validate your Python script, validate your user-defined function before invoking the embedded Python APIs. You'll do this using a script that builds a linear regression model.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_1609796590","id":"20210302-002652_2011518809","dateCreated":"2021-01-11T19:10:25+0000","dateStarted":"2021-03-02T19:23:13+0000","dateFinished":"2021-03-02T19:23:13+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:45"},{"text":"%md\n.\nRun the following script to create a linear model with sklearn.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:13+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Run the following script to create a linear model with sklearn.
.\n
Predict the petal length using the predict function.
.\n
Assess model quality using mean squared error and R^2.
.\n
Generate a scatterplot of the data along with a plot of the regression line.
In this step, you will build the same linear model, but using the embedded Python execution Python engines under control of the Autonomous Database environment. You will perform the following tasks:
\nNote, in the Python API, the embedded Python execution functions accept a function object or named string as input, and can return images as well as structured content.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_324731957","id":"20210302-002652_1512000101","dateCreated":"2021-01-11T19:22:58+0000","dateStarted":"2021-03-02T19:23:20+0000","dateFinished":"2021-03-02T19:23:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:54"},{"text":"%md\n.\nDefine the user-defined function *build_lm_1*.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:20+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the user-defined function build_lm_1.
.\n
Invoke your user-defined function to build the model and plot the petal length predictions.
oml.table_apply takes the OML DataFrame proxy object IRIS as input data and loads that data to the user-defined function as a pandas DataFrame. In this example, the user-defined function is passed as a Python string. You see that the model comes back as an OML object, which you can pull to the client to view the the linear model.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-212806346","id":"20210302-002652_1953309171","dateCreated":"2021-01-11T19:31:10+0000","dateStarted":"2021-03-02T19:23:25+0000","dateFinished":"2021-03-02T19:23:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:59"},{"text":"%python\n\nimport inspect\n\nbuild_lm_1 = \"\"\"def build_lm_1(dat):\n import oml\n from sklearn import linear_model\n import matplotlib.pyplot as plt\n\n lm = linear_model.LinearRegression()\n X = dat[[\"PETAL_WIDTH\"]]\n y = dat[[\"PETAL_LENGTH\"]]\n mod = lm.fit(X, y)\n\n pred = mod.predict(dat[[\"PETAL_WIDTH\"]])\n plt.scatter(dat.loc[:,\"PETAL_WIDTH\"], dat.loc[:,\"PETAL_LENGTH\"])\n plt.plot(dat[[\"PETAL_WIDTH\"]], pred, color='blue', linewidth=3)\n plt.grid(True)\n plt.title('Prediction of Petal Length')\n\n plt.xlabel('Petal Width')\n plt.ylabel('Petal Length')\n plt.show()\n return mod\"\"\"","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:25+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812115_1141787690","id":"20210302-002652_697575281","dateCreated":"2021-01-11T19:40:26+0000","dateStarted":"2021-03-02T19:23:25+0000","dateFinished":"2021-03-02T19:23:26+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:60"},{"text":"%md\n.\nBy invoking **table_apply**, a Python engine is spawned and the user-defined function *build_lm_1* is invoked on that engine with the data referenced by *IRIS* being passed in as a pandas DataFrame. Part of the return value is the image, which is automatically displayed. In this example, we are passing the function object to the ***table_apply*** function.\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:26+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
By invoking table_apply, a Python engine is spawned and the user-defined function build_lm_1 is invoked on that engine with the data referenced by IRIS being passed in as a pandas DataFrame. Part of the return value is the image, which is automatically displayed. In this example, we are passing the function object to the table_apply function.
Use the embedded Python execution function oml.row_apply to run a user-defined function on chunks of rows, which is useful to perform scoring in parallel for native Python models. In this example, we pass the function to row_apply as a Python function object.
\nDefine the user-defined function score_lm_1 to make predictions (score data) using the data set and model passed in as arguments. It returns the predictions as a DataFrame object.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-432540894","id":"20210302-002652_2111088335","dateCreated":"2021-01-11T19:46:53+0000","dateStarted":"2021-03-02T19:23:30+0000","dateFinished":"2021-03-02T19:23:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:64"},{"text":"%python\n\ndef score_lm_1(dat, model):\n import pandas as pd\n from sklearn import linear_model\n pred = model.predict(dat[[\"PETAL_WIDTH\"]])\n return pd.concat([dat[['SPECIES', 'PETAL_LENGTH']], \n pd.DataFrame(pred, columns=['PRED_PETAL_LENGTH'])], axis=1)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:30+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812115_-406771513","id":"20210302-002652_942174813","dateCreated":"2021-01-11T19:47:49+0000","dateStarted":"2021-03-02T19:23:30+0000","dateFinished":"2021-03-02T19:23:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:65"},{"text":"%md\n.\nUse ***row_apply*** to invoke this user-defined function and return a single OML DataFrame proxy object as the result. The ***row_apply*** function takes as arguments the proxy object *IRIS*, that we want 10 rows scored at a time (resulting in 15 function invocations), the user-defined function, the linear model object, and that we want the result to be returned as a single table by specifying the table definition.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:30+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use row_apply to invoke this user-defined function and return a single OML DataFrame proxy object as the result. The row_apply function takes as arguments the proxy object IRIS, that we want 10 rows scored at a time (resulting in 15 function invocations), the user-defined function, the linear model object, and that we want the result to be returned as a single table by specifying the table definition.
This step shows how to use the oml.group_apply function for model building. The group_apply apply function passes the OML DataFrame specified by the data argument to the user-defined func function as its first argument. The index argument specifies the columns of oml.DataFrame by which the database groups the data for processing by the user-defined Python function. The group_apply function can use data-parallel execution, in which one or more Python engines perform the same Python function on different groups of data.
\nIn this step, you build three models, one specific to each species and return them as a dictionary. These models are indexed by their corresponding index value.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_632144300","id":"20210302-002652_54298503","dateCreated":"2021-01-11T19:51:46+0000","dateStarted":"2021-03-02T19:23:35+0000","dateFinished":"2021-03-02T19:23:35+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:68"},{"text":"%python\n\ndef build_lm_g(dat):\n import oml\n from sklearn import linear_model\n lm = linear_model.LinearRegression()\n X = dat[[\"PETAL_WIDTH\"]]\n y = dat[[\"PETAL_LENGTH\"]]\n mod = lm.fit(X, y)\n return mod\n\nmod = oml.group_apply(IRIS[:,[\"PETAL_LENGTH\",\"PETAL_WIDTH\",\"SPECIES\"]], \n index=oml.DataFrame(IRIS['SPECIES']), \n func=build_lm_g, \n parallel=2)\n \nprint(\"Type:\",type(mod))\nmod\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:35+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Type:.\n
Modify the user-defined function to save the resulting model in the same named datastore. A datastore allows storing Python objects in the database under a provided name. The stored Python object assumes the variable name assigned in the Python environment. Here, we construct a name concatenating mod_ as a prefix and the corresponding Species value.
.\n
Use group_apply to invoke the user-defined function and list the resulting models, which are a dictionary of three elements each assigned the model object name. The group_apply function takes the data, the index parameter that specifies the column or columns to partition on, the user-defined function, and that we wish to automatically connect to the database from the Python engine. Connecting to the database is necessary when using the datastore functionality.
Note, if the datastore exists, then delete it so that the group_apply function completes successfully.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-1419333425","id":"20210302-002652_14357513","dateCreated":"2021-01-11T19:59:30+0000","dateStarted":"2021-03-02T19:23:39+0000","dateFinished":"2021-03-02T19:23:39+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:72"},{"text":"%python\n\ntry:\n oml.ds.delete('ds-1')\nexcept:\n print(\"Datastore not found\")\n\nres = oml.group_apply(IRIS[:,[\"PETAL_LENGTH\",\"PETAL_WIDTH\",\"SPECIES\"]], \n index=oml.DataFrame(IRIS['SPECIES']), \n func=build_lm_2, dsname=\"ds-1\")\n\nprint(\"Outcome:\",res)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:39+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Outcome: {'virginica': 'mod_virginica', 'setosa': 'mod_setosa', 'versicolor': 'mod_versicolor'}\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_1299417018","id":"20210302-002652_361422778","dateCreated":"2021-01-11T20:00:33+0000","dateStarted":"2021-03-02T19:23:39+0000","dateFinished":"2021-03-02T19:23:45+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:73"},{"text":"%md\n.\nHere, the model object names are *mod_versicolor*, *mod_virginica*, and *mod_setosa*. When you load the datastore, you get the three models loaded into the client Python engine, assigned to their respective variables.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:45+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Here, the model object names are mod_versicolor, mod_virginica, and mod_setosa. When you load the datastore, you get the three models loaded into the client Python engine, assigned to their respective variables.
Use the group_apply function to count the number of each species in the data set.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_281288725","id":"20210302-002652_124863513","dateCreated":"2021-01-11T20:06:56+0000","dateStarted":"2021-03-02T19:23:46+0000","dateFinished":"2021-03-02T19:23:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:75"},{"text":"%python\n\n# Insert code here\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:46+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812115_1851522511","id":"20210302-002652_1607623586","dateCreated":"2021-01-11T20:07:51+0000","dateStarted":"2021-03-02T19:23:46+0000","dateFinished":"2021-03-02T19:23:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:76"},{"title":"Step 5: Invoke a function N times","text":"%md\n\nThe ***index_apply*** function allows the same function to be invoked a specified number of times. The first argument to the user-defined function is an index number for the function execution. For example, if the ***times*** argument is 10, each function invocation will receive a distinct value between 1 and 10. This can be used, e.g., for selecting behavior within the function or setting a random seed for Monte Carlo analysis.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"The index_apply function allows the same function to be invoked a specified number of times. The first argument to the user-defined function is an index number for the function execution. For example, if the times argument is 10, each function invocation will receive a distinct value between 1 and 10. This can be used, e.g., for selecting behavior within the function or setting a random seed for Monte Carlo analysis.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_41349087","id":"20210302-002652_484835565","dateCreated":"2021-01-11T20:05:15+0000","dateStarted":"2021-03-02T19:23:47+0000","dateFinished":"2021-03-02T19:23:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:77"},{"text":"%python\n\ndef compute_random_mean(index):\n import numpy as np\n import scipy\n from statistics import mean \n np.random.seed(index)\n res = np.random.random((100,1))*10\n return mean(res[1])\n \nprint(\"Test: \", compute_random_mean(3))\n\nres = oml.index_apply(times=12, func=compute_random_mean)\nres\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:47+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Test: 7.081478226181048\n[7.203244934421581, 0.25926231827891333, 7.081478226181048, 5.4723224917572235, 8.707323061773764, 3.3197980530117723, 7.7991879224011464, 9.68540662820932, 5.018745921487388, 0.207519493594015, 0.19475241487624584, 7.400496965154048]\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-1049996132","id":"20210302-002652_328670438","dateCreated":"2021-01-11T20:06:34+0000","dateStarted":"2021-03-02T19:23:48+0000","dateFinished":"2021-03-02T19:23:48+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:78"},{"title":"Step 6: Return multiple images from Embedded Python Execution","text":"%md\n\nThis step shows how to create a function *RandomRedDots* that creates and returns a simple DataFrame and generates two scatterplots of random red dots. First, create *RandomRedDots* and save it in the Python Script Repository, and then define and run the native Python function.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:48+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"This step shows how to create a function RandomRedDots that creates and returns a simple DataFrame and generates two scatterplots of random red dots. First, create RandomRedDots and save it in the Python Script Repository, and then define and run the native Python function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-861244375","id":"20210302-002652_508285697","dateCreated":"2021-01-11T20:11:27+0000","dateStarted":"2021-03-02T19:23:49+0000","dateFinished":"2021-03-02T19:23:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:79"},{"text":"%python\n\nRandomRedDots = \"\"\"def RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df\"\"\"\n\n\noml.script.create(\"RandomRedDots\", func=RandomRedDots, is_global=True, overwrite=True)\n\nprint(RandomRedDots)\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:49+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined","editorHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"def RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-681100817","id":"20210302-002652_1388866585","dateCreated":"2021-01-11T20:38:19+0000","dateStarted":"2021-03-02T19:23:49+0000","dateFinished":"2021-03-02T19:23:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:80"},{"text":"%python\n\ndef RandomRedDots (num_dots_1 = 100, num_dots_2 = 10):\n import numpy as np\n import pandas as pd\n import matplotlib.pyplot as plt\n\n d = {'id': range(1,10), 'val': [x/100 for x in range(1,10)]}\n df = pd.DataFrame(data=d)\n plt.scatter(range(0,num_dots_1), np.random.rand(100),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n plt.scatter(range(0,num_dots_2), np.random.rand(10),c='r')\n plt.title(\"Random Red Dots\")\n plt.show()\n return df","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:50+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812115_-167802659","id":"20210302-002652_2122706885","dateCreated":"2021-01-18T04:06:24+0000","dateStarted":"2021-03-02T19:23:50+0000","dateFinished":"2021-03-02T19:23:50+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:81"},{"text":"%python\n\nRandomRedDots()","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:50+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the oml.do_eval function to invoke the function RandomRedDots. The do_eval function does not automatically pass data to the function, it merely invokes it. Here, we show accepting the default argument values for RandomRedDots.
.\n
In this next example, we modify the function to use subplots, thereby creating separate figure objects for the scatter plots. Store this in the script repository as RandomRedDots2 and invoke the function to see the results. As expected, you get both plots.
Note, when you call RandomRedDots2 using embedded Python execution, you will get both plots as shown in the result.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_596172479","id":"20210302-002652_1440648246","dateCreated":"2021-01-18T04:05:28+0000","dateStarted":"2021-03-02T19:23:55+0000","dateFinished":"2021-03-02T19:23:55+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:85"},{"text":"%md\n.\nDefine the *RandomRedDots2* function that generates two scatter plots, and returns a 2 column DataFrame. Note that we can pass arguments to these functions, here, *num_dots_1* and *num_dots_2*.","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:55+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the RandomRedDots2 function that generates two scatter plots, and returns a 2 column DataFrame. Note that we can pass arguments to these functions, here, num_dots_1 and num_dots_2.
.\n
Use the oml.do_eval function to call the function RandomRedDots2. In this invocation, we're specifying arguments to do_eval for num_dots_1 and num_dots_2. These are specified as you would any other argument to do_eval. This applies to the other embedded Python functions as well.
OML4Py stores named user-defined functions called scripts in the script repository. You can make scripts either private or global. A private script is available only to the owner. A global script is available to any user. For private scripts, the owner of the script may grant the read privilege to other users or revoke that privilege.
\nTo illustrate using the Python Script Repository, you will define a function build_lm_str that will fit a regression model. Using this function, you will then create a script named MyLM_function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-355652399","id":"20210302-002652_1268275665","dateCreated":"2021-01-18T01:54:21+0000","dateStarted":"2021-03-02T19:23:59+0000","dateFinished":"2021-03-02T19:23:59+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:90"},{"text":"%md\n.\nTo store a user-defined function in the script repository, it must be presented as a named string. Define the function as a string, *build_lm_str*. Note the use of triple quotes to enable formatting. ","user":"OMLUSER02","dateUpdated":"2021-03-02T19:23:59+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
To store a user-defined function in the script repository, it must be presented as a named string. Define the function as a string, build_lm_str. Note the use of triple quotes to enable formatting.
.\n
View the string that you just created.
Use the function oml.script.create to create a script MyLM_function.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-353403783","id":"20210302-002652_1900763376","dateCreated":"2021-01-18T02:10:27+0000","dateStarted":"2021-03-02T19:24:03+0000","dateFinished":"2021-03-02T19:24:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:95"},{"text":"%md\n.\nYou can use the ***oml.script.drop*** function to drop a named script, here shown in a try-except block. Then, use the ***oml.script.create*** function to create the script *MyLM_function*. Note that we have the *overwrite* argument set to *True* in case the function already exists.","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:03+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
You can use the oml.script.drop function to drop a named script, here shown in a try-except block. Then, use the oml.script.create function to create the script MyLM_function. Note that we have the overwrite argument set to True in case the function already exists.
List the scripts that are available only to the current user with oml.script.dir with sctype set to user.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_1031024361","id":"20210302-002652_870686018","dateCreated":"2021-01-18T02:14:33+0000","dateStarted":"2021-03-02T19:24:05+0000","dateFinished":"2021-03-02T19:24:05+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:99"},{"text":"%python\n\noml.script.dir(sctype='user')\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:06+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" name ... date\n0 MyLM_function ... 2021-03-02 19:24:05\n1 MyTEST_function ... 2021-03-02 01:13:23\n\n[2 rows x 4 columns]\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-1977845062","id":"20210302-002652_1681436207","dateCreated":"2021-01-18T02:15:16+0000","dateStarted":"2021-03-02T19:24:06+0000","dateFinished":"2021-03-02T19:24:06+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:100"},{"text":"%md\n.\nGrant the read privilege on the *MyLM_function* script to user OMLUSER with ***oml.grant***. Notice that the type is specified as *pyqscript*. This same function is used to grant privileges on datastores. \n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:06+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Grant the read privilege on the MyLM_function script to user OMLUSER with oml.grant. Notice that the type is specified as pyqscript. This same function is used to grant privileges on datastores.
.\n
List the scripts to which the read privilege has been granted using oml.script.dir and sctype set to grant.
.\n
Load the named function into the Python engine for use as a typical Python function using oml.script.load.
.\n
Extract the function text string from the function object and use this to save in the script repository using oml.script_create.
Use the function oml.script.dir to list all the available scripts.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_909356291","id":"20210302-002652_607147080","dateCreated":"2021-01-18T02:19:45+0000","dateStarted":"2021-03-02T19:24:14+0000","dateFinished":"2021-03-02T19:24:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:110"},{"text":"%python\n\noml.script.dir(sctype=\"all\")\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:14+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":" owner name ... description date\n0 OMLUSER02 MyLM_function ... None 2021-03-02 19:24:05\n1 OMLUSER02 MyTEST_function ... None 2021-03-02 19:24:13\n2 PYQSYS RandomRedDots ... None 2021-03-02 19:23:49\n3 PYQSYS RandomRedDots2 ... None 2021-03-02 19:23:56\n\n[4 rows x 5 columns]\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_1105985058","id":"20210302-002652_449414693","dateCreated":"2021-01-18T02:20:01+0000","dateStarted":"2021-03-02T19:24:14+0000","dateFinished":"2021-03-02T19:24:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:111"},{"text":"%md\n.\nInvoke ***table_apply*** on *build_lm_str* and *loaded_str*. Recall that these strings represent the same function - *build_lm_str* was saved to the script repository after assigning the function to a string object. *loaded_str* is the string representation of the function extracted using ***get_source().read()***.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:14+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Invoke table_apply on build_lm_str and loaded_str. Recall that these strings represent the same function - build_lm_str was saved to the script repository after assigning the function to a string object. loaded_str is the string representation of the function extracted using get_source().read().
A global function is one that can be accessed by any user.
\nIn this step, you will define and save a global function build_lm3. You will then call function build_lm3 to build another model.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812115_-930880299","id":"20210302-002652_2114847910","dateCreated":"2021-01-18T02:20:18+0000","dateStarted":"2021-03-02T19:24:17+0000","dateFinished":"2021-03-02T19:24:17+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:115"},{"text":"%md\n.\nDefine the function *build_lm3* as a named string and save it to the script repository.","user":"OMLUSER02","dateUpdated":"2021-03-02T19:24:18+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the function build_lm3 as a named string and save it to the script repository.
.\n
Use the oml.table_apply function to invoke this function with the IRIS table. The oml.table_apply function spawns a Python engine, loads the specified user-defined function, and passes the data referenced by the IRIS proxy object as a pandas DataFrame in the first argument of the user-defined function. The resulting scikit-learn linear model is returned in res.
In this step, you will perform the following:
\n.\n
First drop the private script MyLM_function, then drop the global script MyGlobalML_function, and finally list the available scripts.
Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nAutoML (Automatic Machine Learning) provides built-in data science expertise about data analytics and modeling that you can employ to build machine learning models.
\nIn this lab, we highlight the AutoML workflow using OML4Py:
\nCopyright (c) 2021 Oracle Corporation
\nImport the oml package, Pandas package, and automl module.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_-1507520381","id":"20210224-211339_742285688","dateCreated":"2021-01-11T21:41:23+0000","dateStarted":"2021-02-24T21:56:58+0000","dateFinished":"2021-02-24T21:56:58+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:23"},{"text":"%python\n\nimport pandas as pd\nimport oml\nfrom oml import automl\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:56:58+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219696_1148333566","id":"20210224-211339_767246765","dateCreated":"2021-01-11T21:46:41+0000","dateStarted":"2021-02-24T21:56:59+0000","dateFinished":"2021-02-24T21:57:07+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:24"},{"text":"%md\n.\nUse the ***oml.sync*** function to create an OML Dataframe as a proxy for the database table *WINE*.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:57:07+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the oml.sync function to create an OML Dataframe as a proxy for the database table WINE.
Prepare the wine data set by separating predictors from the target as conventional for Python model building. This produces two new proxy objects that will be used in AutoML functions.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_-1683593003","id":"20210224-211339_1355033705","dateCreated":"2021-01-11T21:50:51+0000","dateStarted":"2021-02-24T21:57:09+0000","dateFinished":"2021-02-24T21:57:09+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:27"},{"text":"%python\n\nWINE_X_cl,WINE_y_cl = WINE.drop('target'), WINE['target']\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:57:09+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219696_690437826","id":"20210224-211339_663075317","dateCreated":"2021-01-11T23:29:52+0000","dateStarted":"2021-02-24T21:57:10+0000","dateFinished":"2021-02-24T21:57:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:28"},{"text":"%md\n.\nUse AutoML to select the top classification algorithms for predicting the WINE data target. Display the top ranked algorithms and their accuracy. SVM Gaussian is ranked first and we'll use that in subsequent AutoML function invocations.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:57:10+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use AutoML to select the top classification algorithms for predicting the WINE data target. Display the top ranked algorithms and their accuracy. SVM Gaussian is ranked first and we'll use that in subsequent AutoML function invocations.
In this step, you determine the features that best support the selected algorithm. First define a FeatureSelection object with score metric accuracy and then invoke the reduce function and specify the desired algorithm, in this case as determined above and stored in variable selected_wine_alg_cl. Also specify the train and test OML DataFrame proxy objects.
\nThe set of selected columns is returned.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_465239578","id":"20210224-211339_1239787956","dateCreated":"2021-01-11T23:31:05+0000","dateStarted":"2021-02-24T21:58:29+0000","dateFinished":"2021-02-24T21:58:29+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:31"},{"text":"%md\n.\nDefine the Feature Selection object *fs_wine_cl* and invoke reduce with the selected algorithms and *WINE* proxy objects.\n","user":"OMLUSER","dateUpdated":"2021-02-24T21:58:29+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the Feature Selection object fs_wine_cl and invoke reduce with the selected algorithms and WINE proxy objects.
Try other algorithms, such as svm_linear or rf in the first argument of the reduce function see if different columns are selected.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_674874794","id":"20210224-211339_1247472968","dateCreated":"2021-01-11T23:35:26+0000","dateStarted":"2021-02-24T22:00:50+0000","dateFinished":"2021-02-24T22:00:50+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:34"},{"text":"%python\n\n# Insert code here\n\n\n\n","user":"OMLUSER","dateUpdated":"2021-02-24T22:00:50+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614201219696_-1567681544","id":"20210224-211339_781604286","dateCreated":"2021-01-11T23:39:32+0000","dateStarted":"2021-02-24T22:00:50+0000","dateFinished":"2021-02-24T22:00:50+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:35"},{"title":"Step 4: Automatic model tuning","text":"%md\n\nAt this point, you are ready to build and tune the models you want to use.\n\nFirst, define a ModelTuning object for classification. Then invoke tune to produce the tuned model using the algorithm selected above and the reduced data features.\n\nModel tuning returns a dictionary with the best model and the evaluation results of the other models tried. This also contains a list of the hyperparameter choices tried and their corresponding score.\n","user":"OMLUSER","dateUpdated":"2021-02-24T22:00:50+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"At this point, you are ready to build and tune the models you want to use.
\nFirst, define a ModelTuning object for classification. Then invoke tune to produce the tuned model using the algorithm selected above and the reduced data features.
\nModel tuning returns a dictionary with the best model and the evaluation results of the other models tried. This also contains a list of the hyperparameter choices tried and their corresponding score.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_-430670013","id":"20210224-211339_1115956635","dateCreated":"2021-01-11T23:39:44+0000","dateStarted":"2021-02-24T22:00:51+0000","dateFinished":"2021-02-24T22:00:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:36"},{"text":"%md\n.\nDefine the model tuning object *mt_wine_cl* for classification and invoke tune.\n","user":"OMLUSER","dateUpdated":"2021-02-24T22:00:51+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the model tuning object mt_wine_cl for classification and invoke tune.
.\n
List the hyperparameters and their values tried for the top two models, along with the corresponding model's score metric value.
.\n
Users can also specify a custom search space for model building using the param_space argument to the tune function.
Model selection automatically selects the best algorithm (using Algorithm Selection) from the set of supported algorithms, then builds, tunes, and returns the best model for your data. To use model selection, specify your data and the number of algorithms you want to tune.
\n"}]},"interrupted":false,"jobName":"paragraph_1614201219696_-586720937","id":"20210224-211339_137986738","dateCreated":"2021-01-12T00:23:03+0000","dateStarted":"2021-02-24T22:10:58+0000","dateFinished":"2021-02-24T22:10:59+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:43"},{"text":"%md\n.\nDefine a *ModelSelection* object and invoke select for automatically building the best model on our wine data.","user":"OMLUSER","dateUpdated":"2021-02-24T22:10:59+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define a ModelSelection object and invoke select for automatically building the best model on our wine data.
Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automated machine learning capabilities.
\nAutoML (Automated Machine Learning) provides built-in data science expertise about data analytics and modeling that you can employ to build machine learning models.
\nIn this lab, we highlight the AutoML workflow using OML4Py:
\nCopyright (c) 2021 Oracle Corporation
\nImport the oml package, Pandas package, and automl module.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_1679706415","id":"20210302-002652_2058727932","dateCreated":"2021-01-11T21:41:23+0000","dateStarted":"2021-03-02T19:25:49+0000","dateFinished":"2021-03-02T19:25:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:23"},{"text":"%python\n\nimport pandas as pd\nimport oml\nfrom oml import automl\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:25:49+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812164_-5224215","id":"20210302-002652_2097139835","dateCreated":"2021-01-11T21:46:41+0000","dateStarted":"2021-03-02T19:25:49+0000","dateFinished":"2021-03-02T19:25:58+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:24"},{"text":"%md\n.\nUse the ***oml.sync*** function to create an OML Dataframe as a proxy for the database table *WINE*.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:25:58+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use the oml.sync function to create an OML Dataframe as a proxy for the database table WINE.
Prepare the wine data set by separating predictors from the target as conventional for Python model building. This produces two new proxy objects that will be used in AutoML functions.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_-563538432","id":"20210302-002652_1110950954","dateCreated":"2021-01-11T21:50:51+0000","dateStarted":"2021-03-02T19:26:01+0000","dateFinished":"2021-03-02T19:26:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:27"},{"text":"%python\n\nWINE_X_cl,WINE_y_cl = WINE.drop('target'), WINE['target']\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:26:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812164_856065807","id":"20210302-002652_1078568991","dateCreated":"2021-01-11T23:29:52+0000","dateStarted":"2021-03-02T19:26:01+0000","dateFinished":"2021-03-02T19:26:01+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:28"},{"text":"%md\n.\nUse AutoML to select the top classification algorithms for predicting the WINE data target. Display the top ranked algorithms and their accuracy. SVM Gaussian is ranked first and we'll use that in subsequent AutoML function invocations.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:26:01+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Use AutoML to select the top classification algorithms for predicting the WINE data target. Display the top ranked algorithms and their accuracy. SVM Gaussian is ranked first and we'll use that in subsequent AutoML function invocations.
In this step, you determine the features that best support the selected algorithm. First define a FeatureSelection object with score metric accuracy and then invoke the reduce function and specify the desired algorithm, in this case as determined above and stored in variable selected_wine_alg_cl. Also specify the train and test OML DataFrame proxy objects.
\nThe set of selected columns is returned.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_1188502969","id":"20210302-002652_1947358098","dateCreated":"2021-01-11T23:31:05+0000","dateStarted":"2021-03-02T19:27:35+0000","dateFinished":"2021-03-02T19:27:35+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:32"},{"text":"%md\n.\nDefine the Feature Selection object *fs_wine_cl* and invoke reduce with the selected algorithms and *WINE* proxy objects.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:27:35+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the Feature Selection object fs_wine_cl and invoke reduce with the selected algorithms and WINE proxy objects.
Try other algorithms, such as svm_linear or rf in the first argument of the reduce function see if different columns are selected.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_-152459467","id":"20210302-002652_1719185936","dateCreated":"2021-01-11T23:35:26+0000","dateStarted":"2021-03-02T19:29:27+0000","dateFinished":"2021-03-02T19:29:27+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:35"},{"text":"%python\n\n# Insert code here\n\n\n\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:29:27+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"text","editOnDblClick":false},"editorMode":"ace/mode/undefined"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[]},"interrupted":false,"jobName":"paragraph_1614644812164_-694870740","id":"20210302-002652_9495438","dateCreated":"2021-01-11T23:39:32+0000","dateStarted":"2021-03-02T19:29:27+0000","dateFinished":"2021-03-02T19:29:27+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:36"},{"title":"Step 4: Automated model tuning","text":"%md\n\nAt this point, you are ready to build and tune the models you want to use.\n\nFirst, define a ModelTuning object for classification. Then invoke tune to produce the tuned model using the algorithm selected above and the reduced data features.\n\nModel tuning returns a dictionary with the best model and the evaluation results of the other models tried. This also contains a list of the hyperparameter choices tried and their corresponding score.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:29:27+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true,"title":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"At this point, you are ready to build and tune the models you want to use.
\nFirst, define a ModelTuning object for classification. Then invoke tune to produce the tuned model using the algorithm selected above and the reduced data features.
\nModel tuning returns a dictionary with the best model and the evaluation results of the other models tried. This also contains a list of the hyperparameter choices tried and their corresponding score.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_-1491034261","id":"20210302-002652_483625605","dateCreated":"2021-01-11T23:39:44+0000","dateStarted":"2021-03-02T19:29:28+0000","dateFinished":"2021-03-02T19:29:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:37"},{"text":"%md\n.\nDefine the model tuning object *mt_wine_cl* for classification and invoke tune.\n","user":"OMLUSER02","dateUpdated":"2021-03-02T19:29:28+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define the model tuning object mt_wine_cl for classification and invoke tune.
.\n
List the hyperparameters and their values tried for the top two models, along with the corresponding model's score metric value.
.\n
Users can also specify a custom search space for model building using the param_space argument to the tune function.
Model selection automates selecting the best algorithm (using Algorithm Selection) from the set of supported algorithms, then builds, tunes, and returns the best model for your data. To use model selection, specify your data and the number of algorithms you want to tune.
\n"}]},"interrupted":false,"jobName":"paragraph_1614644812164_-41364606","id":"20210302-002652_413390848","dateCreated":"2021-01-12T00:23:03+0000","dateStarted":"2021-03-02T19:40:46+0000","dateFinished":"2021-03-02T19:40:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"commited":true,"$$hashKey":"object:44"},{"text":"%md\n.\nDefine a *ModelSelection* object and invoke **select** to automate building the model on our wine data that performs best given the score metric.","user":"OMLUSER02","dateUpdated":"2021-03-02T19:40:46+0000","config":{"colWidth":12,"fontSize":9,"enabled":true,"results":{},"editorSetting":{"language":"md","editOnDblClick":false},"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":".\n
Define a ModelSelection object and invoke select to automate building the model on our wine data that performs best given the score metric.
Oracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis example contains the answers for the Try it yourself exercises in the OML4Py labs. Note, the code in the paragraphs are not runnable from this notebook. Refer to the lab notebooks for the location where the each exercise will be invoked.
\nCopyright (c) 2021 Oracle Corporation
\nOracle Machine Learning for Python (OML4Py) makes the open source Python scripting language and environment ready for the enterprise and big data. Designed for problems involving both large and small data volumes, OML4Py integrates Python with Oracle Autonomous Database, allowing users to run Python commands and scripts for statistical, machine learning, and visualization analyses on database tables and views using Python syntax. Many familiar Python functions are overloaded that translate Python behavior into SQL for running in-database, as well as new automatic machine learning capabilities.
\nThis example contains the answers for the Try it yourself exercises in the OML4Py labs. Note, the code in the paragraphs are not runnable from this notebook. Refer to the lab notebooks for the location where the each exercise will be invoked.
\nCopyright (c) 2021 Oracle Corporation
\n