.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_tutorial/plot_transformer_discrepancy.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_tutorial_plot_transformer_discrepancy.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_tutorial_plot_transformer_discrepancy.py:


.. _example-transform-discrepancy:

Dealing with discrepancies (tf-idf)
===================================

.. index:: td-idf

`TfidfVectorizer <https://scikit-learn.org/stable/modules/
generated/sklearn.feature_extraction.text.TfidfVectorizer.html>`_
is one transform for which the corresponding converted onnx model
may produce different results. The larger the vocabulary is,
the higher the probability to get different result is.
This example proposes a equivalent model with no discrepancies.

Imports, setups
+++++++++++++++

All imports. It also registered onnx converters for :epkg:`xgboost`
and :epkg:`lightgbm`.

.. GENERATED FROM PYTHON SOURCE LINES 22-60

.. code-block:: Python


    import pprint
    import numpy
    from sklearn.pipeline import Pipeline
    from sklearn.compose import ColumnTransformer
    from sklearn.feature_extraction.text import TfidfVectorizer
    from onnxruntime import InferenceSession
    from skl2onnx import to_onnx


    def print_sparse_matrix(m):
        nonan = numpy.nan_to_num(m)
        mi, ma = nonan.min(), nonan.max()
        if mi == ma:
            ma += 1
        mat = numpy.empty(m.shape, dtype=numpy.str_)
        mat[:, :] = "."
        if hasattr(m, "todense"):
            dense = m.todense()
        else:
            dense = m
        for i in range(m.shape[0]):
            for j in range(m.shape[1]):
                if dense[i, j] > 0:
                    c = int((dense[i, j] - mi) / (ma - mi) * 25)
                    mat[i, j] = chr(ord("A") + c)
        return "\n".join("".join(line) for line in mat)


    def diff(a, b):
        if a.shape != b.shape:
            raise ValueError(
                f"Cannot compare matrices with different shapes {a.shape} != {b.shape}."
            )
        d = numpy.abs(a - b).sum() / a.size
        return d









.. GENERATED FROM PYTHON SOURCE LINES 61-65

Artificial datasets
+++++++++++++++++++

Iris + a text column.

.. GENERATED FROM PYTHON SOURCE LINES 65-82

.. code-block:: Python



    strings = numpy.array(
        [
            "This a sentence.",
            "This a sentence with more characters $^*&'(-...",
            """var = ClassName(var2, user=mail@anywhere.com, pwd"""
            """=")_~-('&]@^\\`|[{#")""",
            "c79857654",
            "https://complex-url.com/;76543u3456?g=hhh&amp;h=23",
            "01-03-05T11:12:13",
            "https://complex-url.com/;dd76543u3456?g=ddhhh&amp;h=23",
        ]
    ).reshape((-1, 1))

    pprint.pprint(strings)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    array([['This a sentence.'],
           ["This a sentence with more characters $^*&'(-..."],
           ['var = ClassName(var2, user=mail@anywhere.com, pwd=")_~-(\'&]@^\\`|[{#")'],
           ['c79857654'],
           ['https://complex-url.com/;76543u3456?g=hhh&amp;h=23'],
           ['01-03-05T11:12:13'],
           ['https://complex-url.com/;dd76543u3456?g=ddhhh&amp;h=23']],
          dtype='<U69')




.. GENERATED FROM PYTHON SOURCE LINES 83-85

Fit a TfIdfVectorizer
+++++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 85-88

.. code-block:: Python


    tfidf = Pipeline([("pre", ColumnTransformer([("tfidf", TfidfVectorizer(), 0)]))])








.. GENERATED FROM PYTHON SOURCE LINES 89-90

We leave a couple of strings out of the training set.

.. GENERATED FROM PYTHON SOURCE LINES 90-98

.. code-block:: Python


    tfidf.fit(strings[:-2])
    tr = tfidf.transform(strings)
    tfidf_step = tfidf.steps[0][1].transformers_[0][1]
    # print(f"output columns: {tfidf_step.get_feature_names_out()}")
    print("rendered outputs")
    print(print_sparse_matrix(tr))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    rendered outputs
    ..............RR.....
    .....M......M.JJ....M
    ...J..JH...J.J...JJJ.
    ....Z................
    JJJ....HJJJ.....J....
    .....................
    K.K....IK.K.....K....




.. GENERATED FROM PYTHON SOURCE LINES 99-101

Conversion to ONNX
++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 101-105

.. code-block:: Python


    onx = to_onnx(tfidf, strings)









.. GENERATED FROM PYTHON SOURCE LINES 106-108

Execution with ONNX
+++++++++++++++++++

.. GENERATED FROM PYTHON SOURCE LINES 108-113

.. code-block:: Python


    sess = InferenceSession(onx.SerializeToString(), providers=["CPUExecutionProvider"])
    got = sess.run(None, {"X": strings})[0]
    print(f"differences={diff(tr, got):g}")
    print(print_sparse_matrix(got))




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    differences=3.25823e-08
    ..............RR.....
    .....M......M.JJ....M
    ...J..JH...J.J...JJJ.
    ....Z................
    JJJ....HJJJ.....J....
    .....................
    K.K....IK.K.....K....





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.039 seconds)


.. _sphx_glr_download_auto_tutorial_plot_transformer_discrepancy.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_transformer_discrepancy.ipynb <plot_transformer_discrepancy.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_transformer_discrepancy.py <plot_transformer_discrepancy.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_