Jekyll2021-06-13T00:26:49+08:00https://blog.arkfeng.xyz/feed.xmlRK.Feng的博客blog for frkhitRK Fengelasticsearch 学习笔记2021-06-12T00:00:00+08:002021-06-12T00:00:00+08:00https://blog.arkfeng.xyz/2021/06/12/elasticsearch_learning_notes<h1 id="elasticsearch-学习笔记">elasticsearch 学习笔记</h1>
<h2 id="1-客户端">1. 客户端</h2>
<h3 id="11-python-客户端">1.1 python 客户端</h3>
<p>安装依赖: <code class="language-plaintext highlighter-rouge">elasticsearch-dsl==7.3.0 elasticsearch==7.12.0</code></p>
<p>客户端连接:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">from</span> <span class="nn">elasticsearch</span> <span class="kn">import</span> <span class="n">AsyncElasticsearch</span>
<span class="n">ES_HOST</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">ES_TIMEOUT</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">60</span>
<span class="n">es_client</span> <span class="o">=</span> <span class="n">AsyncElasticsearch</span><span class="p">(</span><span class="n">hosts</span><span class="o">=</span><span class="p">[</span><span class="n">ES_HOST</span><span class="p">],</span> <span class="n">timeout</span><span class="o">=</span><span class="n">ES_TIMEOUT</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="2-创建索引">2. 创建索引</h2>
<h3 id="21-创建索引">2.1 创建索引</h3>
<p>示例:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PUT demo_index
{
"settings": {
"index": {
"number_of_shards": "3",
"number_of_replicas": "0"
}
},
"mappings": {
"properties": {
"mid": {
"type": "long"
},
"status": {
"type": "integer"
},
"userName": {
"fields": {
"keyword": {
"type": "keyword"
}
},
"type": "text"
},
"tags": {
"properties": {
"tagId": {
"type": "long"
},
"tagName": {
"fields": {
"keyword": {
"type": "keyword"
}
},
"type": "text"
},
"tagType": {
"fields": {
"keyword": {
"type": "keyword"
}
},
"type": "text"
},
"updateTime": {
"type": "long"
}
},
"type": "nested"
}
}
}
}
# 获取当前 mapping
GET /demo_index/_mapping
</code></pre></div></div>
<h3 id="22-字段类型定义">2.2 字段类型定义</h3>
<p>python 环境中定义 模型:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">typing</span> <span class="c1"># noqa
</span>
<span class="kn">from</span> <span class="nn">elasticsearch_dsl</span> <span class="kn">import</span> <span class="n">Document</span><span class="p">,</span> <span class="n">Keyword</span><span class="p">,</span> <span class="n">Long</span><span class="p">,</span> <span class="n">Text</span><span class="p">,</span> <span class="n">Integer</span><span class="p">,</span> <span class="n">Nested</span><span class="p">,</span> <span class="n">InnerDoc</span>
<span class="k">class</span> <span class="nc">Tag</span><span class="p">(</span><span class="n">InnerDoc</span><span class="p">):</span>
<span class="n">tagId</span> <span class="o">=</span> <span class="n">Long</span><span class="p">()</span>
<span class="n">tagName</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s">'keyword'</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">()})</span>
<span class="n">tagType</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s">'keyword'</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">()})</span>
<span class="n">updateTime</span> <span class="o">=</span> <span class="n">Long</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">Model</span><span class="p">(</span><span class="n">Document</span><span class="p">):</span>
<span class="n">mid</span> <span class="o">=</span> <span class="n">Long</span><span class="p">()</span>
<span class="n">status</span> <span class="o">=</span> <span class="n">Integer</span><span class="p">()</span>
<span class="n">userName</span> <span class="o">=</span> <span class="n">Text</span><span class="p">(</span><span class="n">fields</span><span class="o">=</span><span class="p">{</span><span class="s">'keyword'</span><span class="p">:</span> <span class="n">Keyword</span><span class="p">()})</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">Nested</span><span class="p">(</span><span class="n">Tag</span><span class="p">,</span> <span class="n">multi</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</code></pre></div></div>
<p>python 中 可以利用 <code class="language-plaintext highlighter-rouge">Document.init</code> 方法, 直接在代码中创建索引.</p>
<p>更合适的方法时, 通过 Kibana 显式地 使用 <code class="language-plaintext highlighter-rouge">2.1 创建索引</code> 中的 api 创建索引.</p>
<p>一个根据 模型(Document) 创建 api 配置的方法:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">def</span> <span class="nf">build_index_config</span><span class="p">(</span><span class="n">model</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Type</span><span class="p">[</span><span class="n">Document</span><span class="p">],</span> <span class="n">index_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">number_of_shards</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="o">-></span> <span class="nb">dict</span><span class="p">:</span>
<span class="s">""" """</span>
<span class="k">def</span> <span class="nf">create_settings</span><span class="p">(</span><span class="n">model_cls</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Type</span><span class="p">[</span><span class="n">Document</span><span class="p">]):</span>
<span class="n">i</span> <span class="o">=</span> <span class="n">model_cls</span><span class="p">.</span><span class="n">_index</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">i</span><span class="p">.</span><span class="n">to_dict</span><span class="p">()</span>
<span class="k">return</span> <span class="n">config</span>
<span class="k">def</span> <span class="nf">create_index</span><span class="p">(</span><span class="n">_index_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">model_cls</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Type</span><span class="p">[</span><span class="n">Document</span><span class="p">],</span> <span class="n">_number_of_shards</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span>
<span class="n">index_name</span> <span class="o">=</span> <span class="n">_index_name</span>
<span class="n">Index</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="s">"Index"</span><span class="p">,</span> <span class="p">(),</span> <span class="nb">dict</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="n">index_name</span><span class="p">,</span> <span class="n">settings</span><span class="o">=</span><span class="p">{</span><span class="s">"number_of_shards"</span><span class="p">:</span> <span class="n">_number_of_shards</span><span class="p">,</span> <span class="p">}</span>
<span class="p">))</span>
<span class="n">model</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Type</span><span class="p">[</span><span class="n">Document</span><span class="p">]</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">model_cls</span><span class="p">.</span><span class="n">__name__</span><span class="p">,</span> <span class="p">(</span><span class="n">model_cls</span><span class="p">,),</span> <span class="p">{</span>
<span class="s">'Index'</span><span class="p">:</span> <span class="n">Index</span><span class="p">,</span>
<span class="p">})</span>
<span class="n">_index_name</span> <span class="o">=</span> <span class="n">model</span><span class="p">.</span><span class="n">_index</span><span class="p">.</span><span class="n">_name</span>
<span class="k">return</span> <span class="n">create_settings</span><span class="p">(</span><span class="n">model_cls</span><span class="o">=</span><span class="n">model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">create_index</span><span class="p">(</span><span class="n">_index_name</span><span class="o">=</span><span class="n">index_name</span><span class="p">,</span> <span class="n">model_cls</span><span class="o">=</span><span class="n">model</span><span class="p">,</span> <span class="n">_number_of_shards</span><span class="o">=</span><span class="n">number_of_shards</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">build_index_config</span><span class="p">(</span><span class="n">Model</span><span class="p">,</span> <span class="n">index_name</span><span class="o">=</span><span class="s">'demo_index'</span><span class="p">,</span> <span class="n">number_of_shards</span><span class="o">=</span><span class="mi">3</span><span class="p">))</span>
</code></pre></div></div>
<h3 id="23-新增字段">2.3 新增字段</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PUT /demo_index/_mapping
{
"properties": {
"count": {
"type": "integer"
}
}
}
</code></pre></div></div>
<h2 id="3-基本的crud">3. 基本的CRUD</h2>
<h3 id="31-搜索示例">3.1 搜索示例</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 搜索所有
GET /demo_index/_search
{
"query": {
"match_all": {}
},
"_source": ["mid", "userName"],
"size": 100
}
# tags
GET /demo_index/_search
{
"query": {
"nested": {
"path": "tags",
"query": {
"terms": {
"tags.tagName.keyword": [
"商品"
]
}
}
}
}
}
# 范围
GET /demo_index/_search
{
"query": {
"range": {"count":{"gt": 0} }
}
}
GET /demo_index/_search
{
"query": {
"term": {"userName.keyword": "admin" }
}
}
# 排除部分字段
GET demo_index/_search
{
"query": {
"match_all": {}
},
"_source":{
"exclude": ["tags"]
}
}
</code></pre></div></div>
<h3 id="32-更新操作示例">3.2 更新操作示例</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
POST /demo_index/_update_by_query
{
"script": {
"inline": "ctx._source.count = 0",
"lang": "painless"
},
"query": {
"match_all": {}
}
}
POST /demo_index/_update_by_query
{
"script": {
"inline": "ctx._source.tags = []",
"lang": "painless"
},
"query": {
"match_all": {}
}
}
</code></pre></div></div>
<h3 id="33-删除">3.3 删除</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /demo_index/_delete_by_query
{
"query": {
"match_all": {}
}
}
</code></pre></div></div>
<h2 id="4-painless-脚本">4. painless 脚本</h2>
<h2 id="5-python使用实践">5. python使用实践</h2>
<h3 id="51-基本使用">5.1 基本使用</h3>
<p>搜索 <code class="language-plaintext highlighter-rouge">es.search</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">query_dict</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"query"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"match_all"</span><span class="p">:</span> <span class="p">{}</span>
<span class="p">},</span>
<span class="s">"size"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"_source"</span><span class="p">:</span> <span class="p">[</span><span class="s">"tags"</span><span class="p">,</span> <span class="s">"mid"</span><span class="p">]</span>
<span class="p">}</span>
<span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">es_client</span><span class="p">.</span><span class="n">search</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">query_dict</span><span class="p">,</span> <span class="p">)</span>
<span class="k">for</span> <span class="n">doc</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s">"hits"</span><span class="p">][</span><span class="s">"hits"</span><span class="p">]:</span>
<span class="k">yield</span> <span class="n">Model</span><span class="p">(</span><span class="o">**</span><span class="n">doc</span><span class="p">[</span><span class="s">"_source"</span><span class="p">])</span>
</code></pre></div></div>
<p>更新 <code class="language-plaintext highlighter-rouge">es.update_by_query</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">body</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"script"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"source"</span><span class="p">:</span> <span class="s">"ctx._source.count = 0"</span><span class="p">,</span>
<span class="s">"params"</span><span class="p">:</span> <span class="p">{},</span>
<span class="s">"lang"</span><span class="p">:</span> <span class="s">"painless"</span>
<span class="p">},</span>
<span class="s">"query"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"match_all"</span><span class="p">:</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">await</span> <span class="n">es_client</span><span class="p">.</span><span class="n">update_by_query</span><span class="p">(</span><span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span><span class="p">,</span> <span class="n">conflicts</span><span class="o">=</span><span class="s">"proceed"</span><span class="p">)</span>
</code></pre></div></div>
<p>删除 <code class="language-plaintext highlighter-rouge">es.delete_by_query</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">await</span> <span class="n">es_client</span><span class="p">.</span><span class="n">delete_by_query</span><span class="p">(</span>
<span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span><span class="p">,</span>
<span class="n">body</span><span class="o">=</span><span class="p">{</span>
<span class="s">"query"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"bool"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"must"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span><span class="s">"term"</span><span class="p">:</span> <span class="p">{</span><span class="s">"mid"</span><span class="p">:</span> <span class="mi">10</span><span class="p">}}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="n">refresh</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="p">)</span>
</code></pre></div></div>
<h3 id="52-遍历操作">5.2 遍历操作</h3>
<p>使用 <code class="language-plaintext highlighter-rouge">async_scan</code> 实现:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">res_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">doc</span> <span class="ow">in</span> <span class="n">async_scan</span><span class="p">(</span>
<span class="n">client</span><span class="o">=</span><span class="n">es_client</span><span class="p">,</span>
<span class="n">query</span><span class="o">=</span><span class="p">{</span>
<span class="s">"query"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"match_all"</span><span class="p">:</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span>
<span class="p">):</span>
<span class="n">res_list</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">Model</span><span class="p">(</span><span class="o">**</span><span class="n">doc</span><span class="p">[</span><span class="s">"_source"</span><span class="p">]))</span>
</code></pre></div></div>
<p>使用 <code class="language-plaintext highlighter-rouge">search_after</code> 实现:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">query_dict</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"query"</span><span class="p">:</span> <span class="p">{</span>
<span class="s">"match_all"</span><span class="p">:</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">sort_list</span> <span class="o">=</span> <span class="p">[</span><span class="s">"mid:desc"</span><span class="p">]</span> <span class="c1"># 需要提供一个 unique 字段
</span>
<span class="n">raw_query_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">query_dict</span><span class="p">)</span>
<span class="n">_sort</span> <span class="o">=</span> <span class="bp">None</span>
<span class="n">max_size</span> <span class="o">=</span> <span class="mi">200</span>
<span class="n">query_dict</span><span class="p">[</span><span class="s">"size"</span><span class="p">]</span> <span class="o">=</span> <span class="n">max_size</span>
<span class="n">all_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">mid_seen</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">_query_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">query_dict</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_sort</span><span class="p">:</span>
<span class="n">_query_dict</span><span class="p">[</span><span class="s">"search_after"</span><span class="p">]</span> <span class="o">=</span> <span class="n">_sort</span>
<span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">es_client</span><span class="p">.</span><span class="n">search</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">_query_dict</span><span class="p">,</span> <span class="n">sort</span><span class="o">=</span><span class="n">sort_list</span><span class="p">)</span>
<span class="n">_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">doc</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s">"hits"</span><span class="p">][</span><span class="s">"hits"</span><span class="p">]:</span>
<span class="k">yield</span> <span class="n">Model</span><span class="p">(</span><span class="o">**</span><span class="n">doc</span><span class="p">[</span><span class="s">"_source"</span><span class="p">])</span>
<span class="n">_sort</span> <span class="o">=</span> <span class="n">doc</span><span class="p">[</span><span class="s">'sort'</span><span class="p">]</span>
<span class="n">_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">_count</span> <span class="o"><</span> <span class="n">max_size</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">count_info</span> <span class="o">=</span> <span class="k">await</span> <span class="n">es_client</span><span class="p">.</span><span class="n">count</span><span class="p">(</span><span class="n">index</span><span class="o">=</span><span class="s">"demo_index"</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">raw_query_dict</span><span class="p">)</span>
<span class="k">if</span> <span class="n">count_info</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"count"</span><span class="p">)</span> <span class="o">!=</span> <span class="n">all_count</span><span class="p">:</span>
<span class="n">logging</span><span class="p">.</span><span class="n">warning</span><span class="p">(</span><span class="sa">f</span><span class="s">"es.count </span><span class="si">{</span><span class="n">count_info</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'count'</span><span class="p">)</span><span class="si">}</span><span class="s"> != yield_count </span><span class="si">{</span><span class="n">all_count</span><span class="si">}</span><span class="s">!"</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="53-批量操作">5.3 批量操作</h3>RK Fengelasticsearch 学习笔记pandas 备忘2021-06-12T00:00:00+08:002021-06-12T00:00:00+08:00https://blog.arkfeng.xyz/2021/06/12/pandas_notes<h1 id="pandas-备忘">pandas 备忘</h1>
<h2 id="1-读写文件">1. 读写文件</h2>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">def</span> <span class="nf">read_excel</span><span class="p">(</span><span class="n">excel_file</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">only_one_sheet</span><span class="p">:</span> <span class="nb">bool</span><span class="p">)</span> <span class="o">-></span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">:</span>
<span class="n">kwargs</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">only_one_sheet</span><span class="p">:</span>
<span class="n">kwargs</span><span class="p">[</span><span class="s">"sheet_name"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">if</span> <span class="n">excel_file</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">'.'</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="ow">in</span> <span class="p">{</span><span class="s">"xlsx"</span><span class="p">,</span> <span class="s">"xls"</span><span class="p">}:</span>
<span class="k">return</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_excel</span><span class="p">(</span><span class="n">excel_file</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">excel_file</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</code></pre></div></div>
<p>写文件:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">data_list</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="s">"标题 1"</span><span class="p">,</span> <span class="s">"类别 A"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"标题 2"</span><span class="p">,</span> <span class="s">"类别 B"</span><span class="p">),</span>
<span class="p">]</span>
<span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s">"标题"</span><span class="p">,</span> <span class="s">"类别"</span><span class="p">]</span>
<span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span>
<span class="n">data</span><span class="o">=</span><span class="n">data_list</span><span class="p">,</span>
<span class="n">columns</span><span class="o">=</span><span class="n">columns</span>
<span class="p">).</span><span class="n">to_csv</span><span class="p">(</span><span class="s">"文件.csv"</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="2-设置单元格格式">2. 设置单元格格式</h2>
<p>需要安装依赖: <code class="language-plaintext highlighter-rouge">python -m pip install xlsxwriter</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">data_list</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="s">"标题 1"</span><span class="p">,</span> <span class="s">"类别 A"</span><span class="p">),</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"标题 2"</span><span class="p">,</span> <span class="s">"类别 B"</span><span class="p">),</span>
<span class="p">]</span>
<span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s">"index"</span><span class="p">,</span> <span class="s">"标题"</span><span class="p">,</span> <span class="s">"类别"</span><span class="p">]</span>
<span class="k">with</span> <span class="n">pd</span><span class="p">.</span><span class="n">ExcelWriter</span><span class="p">(</span><span class="s">"文件.xlsx"</span><span class="p">)</span> <span class="k">as</span> <span class="n">writer</span><span class="p">:</span>
<span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">(</span>
<span class="n">data</span><span class="o">=</span><span class="n">data_list</span><span class="p">,</span>
<span class="n">columns</span><span class="o">=</span><span class="n">columns</span>
<span class="p">).</span><span class="n">to_excel</span><span class="p">(</span><span class="n">writer</span><span class="p">,</span> <span class="n">sheet_name</span><span class="o">=</span><span class="s">"全部"</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">workbook</span> <span class="o">=</span> <span class="n">writer</span><span class="p">.</span><span class="n">book</span>
<span class="n">basic_format</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">.</span><span class="n">add_format</span><span class="p">()</span>
<span class="n">basic_format</span><span class="p">.</span><span class="n">set_font_size</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span>
<span class="n">basic_format</span><span class="p">.</span><span class="n">set_align</span><span class="p">(</span><span class="s">"top"</span><span class="p">)</span>
<span class="n">title_format</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">.</span><span class="n">add_format</span><span class="p">()</span>
<span class="n">title_format</span><span class="p">.</span><span class="n">set_font_size</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span>
<span class="n">title_format</span><span class="p">.</span><span class="n">set_text_wrap</span><span class="p">()</span>
<span class="n">title_format</span><span class="p">.</span><span class="n">set_align</span><span class="p">(</span><span class="s">"top"</span><span class="p">)</span>
<span class="n">class_format</span> <span class="o">=</span> <span class="n">workbook</span><span class="p">.</span><span class="n">add_format</span><span class="p">()</span>
<span class="n">class_format</span><span class="p">.</span><span class="n">set_font_size</span><span class="p">(</span><span class="mi">12</span><span class="p">)</span>
<span class="n">class_format</span><span class="p">.</span><span class="n">set_text_wrap</span><span class="p">()</span>
<span class="n">class_format</span><span class="p">.</span><span class="n">set_align</span><span class="p">(</span><span class="s">"top"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">sheet_name</span><span class="p">,</span> <span class="n">sheet</span> <span class="ow">in</span> <span class="n">writer</span><span class="p">.</span><span class="n">sheets</span><span class="p">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">sheet</span><span class="p">.</span><span class="n">set_column</span><span class="p">(</span><span class="s">'A:A'</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="n">basic_format</span><span class="p">)</span>
<span class="n">sheet</span><span class="p">.</span><span class="n">set_column</span><span class="p">(</span><span class="s">'B:B'</span><span class="p">,</span> <span class="mi">120</span><span class="p">,</span> <span class="n">title_format</span><span class="p">)</span>
<span class="n">sheet</span><span class="p">.</span><span class="n">set_column</span><span class="p">(</span><span class="s">'C:C'</span><span class="p">,</span> <span class="mi">30</span><span class="p">,</span> <span class="n">class_format</span><span class="p">)</span>
</code></pre></div></div>RK Fengpandas 备忘协程开发下处理 tornado websocket 关闭事件2021-02-10T00:00:00+08:002021-02-10T00:00:00+08:00https://blog.arkfeng.xyz/2021/02/10/websocket_on_close<h1 id="协程开发下处理-tornado-websocket-关闭事件">协程开发下处理 tornado websocket 关闭事件</h1>
<p>验证代码如下:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">logging</span> <span class="c1"># noqa
</span><span class="kn">import</span> <span class="nn">os</span> <span class="c1"># noqa
</span><span class="kn">import</span> <span class="nn">typing</span> <span class="c1"># noqa
</span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span>
<span class="kn">import</span> <span class="nn">arrow</span>
<span class="kn">import</span> <span class="nn">tornado.httpserver</span>
<span class="kn">import</span> <span class="nn">tornado.ioloop</span>
<span class="kn">import</span> <span class="nn">tornado.web</span>
<span class="kn">import</span> <span class="nn">websockets</span>
<span class="kn">from</span> <span class="nn">tornado.websocket</span> <span class="kn">import</span> <span class="n">WebSocketHandler</span>
<span class="k">class</span> <span class="nc">_Ws</span><span class="p">(</span><span class="n">WebSocketHandler</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">check_origin</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">origin</span><span class="p">):</span>
<span class="s">"""校验权限"""</span>
<span class="k">return</span> <span class="bp">True</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">open</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""开启连接"""</span>
<span class="k">print</span><span class="p">(</span><span class="s">"on open"</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">on_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
<span class="s">"""连接通信"""</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"on_message: </span><span class="si">{</span><span class="n">message</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">WsHandler</span><span class="p">(</span><span class="n">_Ws</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">on_close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""关闭连接"""</span>
<span class="k">print</span><span class="p">(</span><span class="s">"on_closed"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">server</span><span class="p">(</span><span class="n">ws_handler_cls</span><span class="p">):</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">set_event_loop</span><span class="p">(</span><span class="n">asyncio</span><span class="p">.</span><span class="n">new_event_loop</span><span class="p">())</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">tornado</span><span class="p">.</span><span class="n">web</span><span class="p">.</span><span class="n">Application</span><span class="p">(</span>
<span class="n">handlers</span><span class="o">=</span><span class="p">[</span>
<span class="p">(</span><span class="sa">r</span><span class="s">'/ws'</span><span class="p">,</span> <span class="n">ws_handler_cls</span><span class="p">),</span>
<span class="p">],</span>
<span class="p">)</span>
<span class="n">app</span><span class="p">.</span><span class="n">listen</span><span class="p">(</span><span class="mi">8004</span><span class="p">)</span>
<span class="n">tornado</span><span class="p">.</span><span class="n">ioloop</span><span class="p">.</span><span class="n">IOLoop</span><span class="p">.</span><span class="n">instance</span><span class="p">().</span><span class="n">start</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">TestWs</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">_req</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="c1"># 这些命令, 有问题
</span> <span class="n">version</span> <span class="o">=</span> <span class="n">arrow</span><span class="p">.</span><span class="n">now</span><span class="p">().</span><span class="n">to</span><span class="p">(</span><span class="s">"local"</span><span class="p">).</span><span class="nb">format</span><span class="p">(</span><span class="s">"YYYY-MM-DD HH:mm:ss"</span><span class="p">)</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">websockets</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span><span class="s">"ws://localhost:8004/ws"</span><span class="p">)</span> <span class="k">as</span> <span class="n">ws</span><span class="p">:</span>
<span class="k">await</span> <span class="n">ws</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="sa">f</span><span class="s">"[</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s">]start"</span><span class="p">)</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">await</span> <span class="n">ws</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="sa">f</span><span class="s">"[</span><span class="si">{</span><span class="n">version</span><span class="si">}</span><span class="s">]stop"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">testBasic</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" 非协程环境下, websocket 能接收到 关闭 事件 """</span>
<span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">server</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">WsHandler</span><span class="p">,),</span> <span class="n">daemon</span><span class="o">=</span><span class="bp">True</span><span class="p">).</span><span class="n">start</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_req</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">testAsyncioBad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" 协程环境下, 使用 async def on_close 无法接收到 关闭 事件 """</span>
<span class="k">class</span> <span class="nc">Ws</span><span class="p">(</span><span class="n">_Ws</span><span class="p">):</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">on_close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""关闭连接"""</span>
<span class="k">print</span><span class="p">(</span><span class="s">"on_closed"</span><span class="p">)</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"print_close"</span><span class="p">)</span>
<span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">server</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">Ws</span><span class="p">,),</span> <span class="n">daemon</span><span class="o">=</span><span class="bp">True</span><span class="p">).</span><span class="n">start</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_req</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">testAsyncioGood</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" 协程环境下, 使用 asyncio.ensure_future 能接收到 关闭 事件 """</span>
<span class="k">class</span> <span class="nc">Ws</span><span class="p">(</span><span class="n">_Ws</span><span class="p">):</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">_close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"print_close"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">on_close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""关闭连接"""</span>
<span class="k">print</span><span class="p">(</span><span class="s">"on_closed"</span><span class="p">)</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">_close</span><span class="p">())</span>
<span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">server</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">Ws</span><span class="p">,),</span> <span class="n">daemon</span><span class="o">=</span><span class="bp">True</span><span class="p">).</span><span class="n">start</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_req</span><span class="p">()</span>
</code></pre></div></div>
<p>结论:</p>
<p>1) 协程环境下, 可以使用 async def get/on_message 等方法, 将接口方法变为 awaitable 方法. 但 on_close 不能使用这个样的方式处理</p>
<p>2) 协程方式执行on_close方法, 需要在方法内部, 使用 <code class="language-plaintext highlighter-rouge">asyncio.ensure_future</code></p>RK Feng协程开发下处理 tornado websocket 关闭事件python多线程编程记录2020-11-07T00:00:00+08:002020-11-07T00:00:00+08:00https://blog.arkfeng.xyz/2020/11/07/python_thread_thinking<h1 id="python多线程编程记录">python多线程编程记录</h1>
<h2 id="1-线程锁额外耗时简单量化">1. 线程锁额外耗时简单量化</h2>
<p>多线程之间, 可借助线程锁共享变量. 线程锁的使用, 会给程序带来多大的性能损耗?</p>
<p>测试示例如下:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">from</span> <span class="nn">enum</span> <span class="kn">import</span> <span class="n">Enum</span>
<span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span><span class="p">,</span> <span class="n">Lock</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">class</span> <span class="nc">BusyMode</span><span class="p">(</span><span class="n">Enum</span><span class="p">):</span>
<span class="n">IO_BUSY</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">CPU_BUSY</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">class</span> <span class="nc">AtomicInteger</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">__slots__</span> <span class="o">=</span> <span class="p">(</span><span class="s">"_value"</span><span class="p">,</span> <span class="s">"_lock"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">0</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="n">value</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_lock</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">inc</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">_lock</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="k">def</span> <span class="nf">dec</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">_lock</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">_lock</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">value</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">_lock</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="k">class</span> <span class="nc">Worker</span><span class="p">(</span><span class="n">Thread</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">count</span><span class="p">:</span> <span class="n">AtomicInteger</span><span class="p">,</span> <span class="n">busy_mode</span><span class="p">:</span> <span class="n">BusyMode</span><span class="p">,</span> <span class="n">times</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">Worker</span><span class="p">,</span> <span class="bp">self</span><span class="p">).</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">count</span> <span class="o">=</span> <span class="n">count</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_busy_mode</span> <span class="o">=</span> <span class="n">busy_mode</span>
<span class="bp">self</span><span class="p">.</span><span class="n">times</span> <span class="o">=</span> <span class="n">times</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">io_busy</span><span class="p">():</span>
<span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">cpu_busy</span><span class="p">():</span>
<span class="k">for</span> <span class="n">count</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1000000</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">count</span> <span class="o">*</span> <span class="n">count</span>
<span class="n">func</span> <span class="o">=</span> <span class="n">io_busy</span> <span class="k">if</span> <span class="bp">self</span><span class="p">.</span><span class="n">_busy_mode</span> <span class="o">==</span> <span class="n">BusyMode</span><span class="p">.</span><span class="n">IO_BUSY</span> <span class="k">else</span> <span class="n">cpu_busy</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">times</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="p">.</span><span class="n">count</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">cur_count</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">count</span><span class="p">.</span><span class="n">inc</span><span class="p">()</span>
<span class="c1"># print("current atomic_count: {}".format(cur_count))
</span> <span class="n">func</span><span class="p">()</span>
<span class="n">result_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_busy_mode</span> <span class="ow">in</span> <span class="p">[</span><span class="n">BusyMode</span><span class="p">.</span><span class="n">IO_BUSY</span><span class="p">,</span> <span class="n">BusyMode</span><span class="p">.</span><span class="n">CPU_BUSY</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">_use_lock</span> <span class="ow">in</span> <span class="p">[</span><span class="bp">False</span><span class="p">,</span> <span class="bp">True</span><span class="p">]:</span>
<span class="k">if</span> <span class="n">_use_lock</span><span class="p">:</span>
<span class="n">_atomic_count</span> <span class="o">=</span> <span class="n">AtomicInteger</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_atomic_count</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">print</span><span class="p">(</span><span class="s">"busy mode {}, use lock {} >>>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">_busy_mode</span><span class="p">,</span> <span class="n">_use_lock</span><span class="p">))</span>
<span class="n">time_start</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">threads</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">20</span><span class="p">):</span>
<span class="n">threads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">Worker</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="n">_atomic_count</span><span class="p">,</span> <span class="n">busy_mode</span><span class="o">=</span><span class="n">_busy_mode</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="mi">20</span><span class="p">))</span>
<span class="k">for</span> <span class="n">worker</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
<span class="n">worker</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">worker</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
<span class="n">worker</span><span class="p">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">time_stop</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">log_line</span> <span class="o">=</span> <span class="s">"busy mode {}, use lock {}, time cost {:.3f}s"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">_busy_mode</span><span class="p">,</span> <span class="n">_use_lock</span><span class="p">,</span> <span class="n">time_stop</span> <span class="o">-</span> <span class="n">time_start</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{}</span><span class="se">\n</span><span class="s">"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">log_line</span><span class="p">))</span>
<span class="n">result_list</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">log_line</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n\n</span><span class="s">result:</span><span class="se">\n</span><span class="s">{}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">result_list</span><span class="p">)))</span>
</code></pre></div></div>
<p>执行结果:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>busy mode BusyMode.IO_BUSY, use lock False, time cost 20.066s
busy mode BusyMode.IO_BUSY, use lock True, time cost 20.056s
busy mode BusyMode.CPU_BUSY, use lock False, time cost 32.659s
busy mode BusyMode.CPU_BUSY, use lock True, time cost 30.786s
</code></pre></div></div>
<h1 id="2-基于多线程-实现-master-worker-模式">2. 基于多线程, 实现 master-worker 模式</h1>
<p>示例:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">typing</span>
<span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">import</span> <span class="nn">weakref</span> <span class="c1"># noqa
</span><span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span>
<span class="k">def</span> <span class="nf">busy_working_by_subprocess</span><span class="p">(</span><span class="n">sleep_time</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="n">subprocess_cmd</span> <span class="o">=</span> <span class="s">"""sleep {}"""</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">sleep_time</span><span class="p">))</span>
<span class="c1"># 环境变量
</span> <span class="n">base_env</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">)</span>
<span class="n">path_dir</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">)</span>
<span class="n">python_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">path_dir</span><span class="p">,</span> <span class="s">"../"</span><span class="p">)</span>
<span class="n">base_env</span><span class="p">.</span><span class="n">update</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">PYTHONIOENCODING</span><span class="o">=</span><span class="s">"utf-8"</span><span class="p">,</span> <span class="n">LANG</span><span class="o">=</span><span class="s">"en_US.UTF-8"</span><span class="p">,</span> <span class="n">PYTHONPATH</span><span class="o">=</span><span class="s">"{}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">python_path</span><span class="p">))))</span>
<span class="n">subprocess_kwargs</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">shell</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="n">base_env</span><span class="p">,</span> <span class="p">)</span>
<span class="n">subprocess</span><span class="p">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">subprocess_cmd</span><span class="p">,</span> <span class="o">**</span><span class="n">subprocess_kwargs</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">TestThread</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">testMasterWorker</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="n">workers</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Thread</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
<span class="n">sleep_time</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
<span class="n">thread_name</span> <span class="o">=</span> <span class="s">"Thread-{}-{}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">)</span>
<span class="n">worker</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">busy_working_by_subprocess</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">sleep_time</span><span class="p">,),</span> <span class="n">name</span><span class="o">=</span><span class="n">thread_name</span><span class="p">)</span>
<span class="n">worker</span><span class="p">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">worker</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="c1"># workers[thread_name] = weakref.proxy(worker)
</span> <span class="c1"># if thread.is_alive():
</span> <span class="c1"># ReferenceError: weakly-referenced object no longer exists
</span> <span class="n">workers</span><span class="p">[</span><span class="n">thread_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">worker</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{} thread start"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">count</span><span class="p">))</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">all_done</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">done_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">thread_name</span><span class="p">,</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">workers</span><span class="p">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">thread</span><span class="p">.</span><span class="n">is_alive</span><span class="p">():</span>
<span class="n">all_done</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">done_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">done_count</span><span class="si">}</span><span class="s">, </span><span class="si">{</span><span class="n">all_done</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">all_done</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"all thread done"</span><span class="p">)</span>
<span class="n">workers</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="k">break</span>
<span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">testMasterWorkerMasterExit</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="n">workers</span><span class="p">:</span> <span class="n">typing</span><span class="p">.</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Thread</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
<span class="n">sleep_time</span> <span class="o">=</span> <span class="mi">60</span>
<span class="n">count</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">count</span><span class="p">):</span>
<span class="n">thread_name</span> <span class="o">=</span> <span class="s">"Thread-{}-{}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">)</span>
<span class="n">worker</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">busy_working_by_subprocess</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">sleep_time</span><span class="p">,),</span> <span class="n">name</span><span class="o">=</span><span class="n">thread_name</span><span class="p">)</span>
<span class="n">worker</span><span class="p">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">worker</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="c1"># workers[thread_name] = weakref.proxy(worker)
</span> <span class="c1"># if thread.is_alive():
</span> <span class="c1"># ReferenceError: weakly-referenced object no longer exists
</span> <span class="n">workers</span><span class="p">[</span><span class="n">thread_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">worker</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{} thread start"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">count</span><span class="p">))</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">all_done</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">done_count</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">thread_name</span><span class="p">,</span> <span class="n">thread</span> <span class="ow">in</span> <span class="n">workers</span><span class="p">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">thread</span><span class="p">.</span><span class="n">is_alive</span><span class="p">():</span>
<span class="n">all_done</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">done_count</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="n">done_count</span><span class="si">}</span><span class="s">, </span><span class="si">{</span><span class="n">all_done</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">all_done</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"all thread done"</span><span class="p">)</span>
<span class="n">workers</span><span class="p">.</span><span class="n">clear</span><span class="p">()</span>
<span class="k">break</span>
<span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># 强制退出: 模拟 master die
</span> <span class="k">print</span><span class="p">(</span><span class="s">"force exit: sys.exit(1)"</span><span class="p">)</span>
<span class="n">sys</span><span class="p">.</span><span class="nb">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># 强制退出后, master 退出, worker 继续执行
</span>
</code></pre></div></div>RK Fengpython多线程编程记录使用parsel处理xml2020-11-05T00:00:00+08:002020-11-05T00:00:00+08:00https://blog.arkfeng.xyz/2020/11/05/python_xml_xpath<h1 id="使用parsel处理xml">使用parsel处理xml</h1>
<h2 id="1-xml解析">1. xml解析</h2>
<p>使用 <code class="language-plaintext highlighter-rouge">parsel</code>包处理解析 xml 数据:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from parsel import Selector
resp = Selector(text=self.xml, type="xml")
for e in resp.xpath('//node[@id="root"]'):
print(e)
</code></pre></div></div>
<h2 id="2-删除-xml-节点">2. 删除 xml 节点</h2>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">parsel</span> <span class="kn">import</span> <span class="n">Selector</span>
<span class="k">class</span> <span class="nc">TestXmlEdit</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">xml</span> <span class="o">=</span> <span class="s">'''<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
<hierarchy>
<node id="root">
<node index="0" id="AAA" bounds="[1, 1]">
<node index="0" id="a" />
<node index="1" id="b" />
</node>
<node index="1" id="AAA" bounds="[1, 1]">
<node index="0" id="A" />
<node index="1" id="B" />
</node>
</node>
</hierarchy>
'''</span>
<span class="k">def</span> <span class="nf">testParselRemove</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">def</span> <span class="nf">iter_node</span><span class="p">(</span><span class="n">_node</span><span class="p">:</span> <span class="n">Selector</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">"node: tag {}, id {}, index {}, child_count {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span>
<span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">.</span><span class="n">tag</span><span class="p">,</span> <span class="n">_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"id"</span><span class="p">),</span> <span class="n">_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"index"</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">)</span>
<span class="p">))</span>
<span class="k">if</span> <span class="n">_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span> <span class="o">==</span> <span class="s">"A"</span><span class="p">:</span>
<span class="k">print</span><span class="p">()</span>
<span class="k">pass</span>
<span class="k">for</span> <span class="n">_sub_index</span><span class="p">,</span> <span class="n">_sub_node</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">[</span><span class="n">_sub_index</span><span class="p">]</span> <span class="ow">is</span> <span class="n">_sub_node</span>
<span class="n">_sub_selector</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">root</span><span class="o">=</span><span class="n">_sub_node</span><span class="p">,</span> <span class="n">namespaces</span><span class="o">=</span><span class="n">_node</span><span class="p">.</span><span class="n">namespaces</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">_node</span><span class="p">.</span><span class="nb">type</span><span class="p">)</span>
<span class="n">iter_node</span><span class="p">(</span><span class="n">_node</span><span class="o">=</span><span class="n">_sub_selector</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">iter_then_remove_node</span><span class="p">(</span><span class="n">_node</span><span class="p">:</span> <span class="n">Selector</span><span class="p">):</span>
<span class="n">_remove_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_sub_node</span> <span class="ow">in</span> <span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">:</span>
<span class="n">_sub_node_id</span> <span class="o">=</span> <span class="n">_sub_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span>
<span class="n">_sub_node_index</span> <span class="o">=</span> <span class="n">_sub_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"index"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_sub_node_id</span> <span class="o">==</span> <span class="s">"AAA"</span> <span class="ow">and</span> <span class="n">_sub_node_index</span> <span class="o">==</span> <span class="s">"0"</span><span class="p">:</span>
<span class="n">_remove_list</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">_sub_node</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_sub_selector</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">root</span><span class="o">=</span><span class="n">_sub_node</span><span class="p">,</span> <span class="n">namespaces</span><span class="o">=</span><span class="n">_node</span><span class="p">.</span><span class="n">namespaces</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">_node</span><span class="p">.</span><span class="nb">type</span><span class="p">)</span>
<span class="n">iter_then_remove_node</span><span class="p">(</span><span class="n">_node</span><span class="o">=</span><span class="n">_sub_selector</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_remove_list</span><span class="p">:</span>
<span class="k">for</span> <span class="n">_sub_node</span> <span class="ow">in</span> <span class="n">_remove_list</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"remove node: tag {}, id {}, index {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span>
<span class="n">_sub_node</span><span class="p">.</span><span class="n">tag</span><span class="p">,</span> <span class="n">_sub_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"id"</span><span class="p">),</span> <span class="n">_sub_node</span><span class="p">.</span><span class="n">attrib</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"index"</span><span class="p">)</span>
<span class="p">))</span>
<span class="n">_node</span><span class="p">.</span><span class="n">root</span><span class="p">.</span><span class="n">remove</span><span class="p">(</span><span class="n">_sub_node</span><span class="p">)</span>
<span class="n">root</span> <span class="o">=</span> <span class="n">Selector</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="bp">self</span><span class="p">.</span><span class="n">xml</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s">"xml"</span><span class="p">)</span>
<span class="n">iter_node</span><span class="p">(</span><span class="n">_node</span><span class="o">=</span><span class="n">root</span><span class="p">)</span>
<span class="n">node_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n\n\n</span><span class="s">before:"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">tree</span> <span class="ow">in</span> <span class="n">root</span><span class="p">.</span><span class="n">xpath</span><span class="p">(</span><span class="s">'//node[@id="AAA"]'</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">tree</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"id"</span><span class="p">],</span> <span class="n">tree</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"bounds"</span><span class="p">])</span>
<span class="n">iter_node</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="n">node_list</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
<span class="c1"># 删除节点
</span> <span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n\n\n</span><span class="s">remove node:"</span><span class="p">)</span>
<span class="n">iter_then_remove_node</span><span class="p">(</span><span class="n">_node</span><span class="o">=</span><span class="n">root</span><span class="p">)</span>
<span class="c1"># root.remove(node_list[0]) # 报错 ValueError: Element is not a child of this node.
</span> <span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n\n\n</span><span class="s">after remove:"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">tree</span> <span class="ow">in</span> <span class="n">root</span><span class="p">.</span><span class="n">xpath</span><span class="p">(</span><span class="s">'//node[@id="AAA"]'</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="n">tree</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"id"</span><span class="p">],</span> <span class="n">tree</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"bounds"</span><span class="p">])</span>
<span class="n">iter_node</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="3-节点搜索举例">3. 节点搜索举例</h2>
<h4 id="多个关系">多个关系</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 节点内
resp.xpath('//node[@text="a" or @text="b"]')
resp.xpath('//node[@text="a" and @class="X"]')
resp.xpath('//node[( @text="a" or @text="b" ) and @class="X"]')
resp.xpath('//node[( @text="a" or @text="b" ) and @class="X" and not(.//node[@class="X"]) ]')
# 跨节点
resp.xpath('//node[@text="a"] | //node[@text="b"]')
</code></pre></div></div>
<h4 id="正则匹配">正则匹配</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>namespaces={'re': "http://exslt.org/regular-expressions"}
resp.xpath('//node[re:test(@class, ".*FrameLayout")]', namespaces=namespaces)
resp.xpath('//a[re:test(@id, "^hypProduct_[0-9]+$")]', namespaces=namespaces)
resp.xpath('//a[re:test(@class, "(?<!EditText)$")]', namespaces=namespaces)
resp.xpath('//node[re:test(@class, ".*A") or re:test(@class, ".*B") ]', namespaces=namespaces)
</code></pre></div></div>
<h4 id="相对位置">相对位置</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 最后一个
resp.xpath('(//node[@text="X")])[last()]')
# A -> 父 -> 父 -> B
resp.xpath((
'//node[@text="A"]'
'/parent::node/parent::node/'
'node[@text="B"]'
))
</code></pre></div></div>RK Feng使用parsel处理xml释放 mac 磁盘空间2020-09-26T00:00:00+08:002020-09-26T00:00:00+08:00https://blog.arkfeng.xyz/2020/09/26/mac_clean_drive<h1 id="释放-mac-磁盘空间">释放 mac 磁盘空间</h1>
<h2 id="1-彻底删除软件">1. 彻底删除软件</h2>
<p><a href="https://www.zhihu.com/question/19551885/answer/229372713">参考: 如何彻底卸载在 Mac 上安装的一个软件? - 新垣結衣的回答 - 知乎</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>通过 关于本机 - 储存空间 - 管理 - 检查文件 - 左边栏中的应用程序按钮来进行删除.
</code></pre></div></div>
<p>同时, 删除应用目录:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rm ~/Library/Application\ Support/<应用目录>
</code></pre></div></div>
<h2 id="磁盘释放流程">磁盘释放流程</h2>
<p>先借助 <code class="language-plaintext highlighter-rouge">关于本机 - 储存空间 - 管理 - 检查文件</code> , 处理大文件.</p>
<p>如果大文件处理完, 还需要释放磁盘, 可以执行:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># 判断 当前用户下, 目录大小
>> cd ~
>> du -sh * | grep G
2.9G Downloads
40G Library
2.1G OneDrive
17G Pictures
6.8G data
# 进入占用磁盘大的目录, 如 Library, 重复执行上述动作
>> cd Library
>> du -sh * | grep G
15G Android
8.1G Application Support
4.5G Caches
8.1G Containers
3.9G Developer
0B GameKit
4.8M Google
95M Group Containers
</code></pre></div></div>RK Feng释放 mac 磁盘空间plantuml使用总结2020-09-19T00:00:00+08:002020-09-19T00:00:00+08:00https://blog.arkfeng.xyz/2020/09/19/plantuml<h1 id="plantuml使用总结">plantuml使用总结</h1>
<h2 id="1-绘制流程图">1. 绘制流程图</h2>
<p>一个基本例子</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@startuml
start
:打开网站;
if (第一次登录) then (yes)
:注册;
else (no)
:直接登录;
endif
repeat :打开新页面;
:执行页面处理;
backward: 处理下一个页面;
repeat while (还有待处理页面?)
stop
@enduml
</code></pre></div></div>
<p><img src="../../../../public/img/plantuml/e1.png" alt="流程图1" /></p>
<p>使用 <code class="language-plaintext highlighter-rouge">partition</code> 定义流程块:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@startuml
start
:打开网站;
partition 登录步骤 {
if (第一次登录) then (yes)
:注册;
else (no)
:直接登录;
endif
}
partition 执行步骤 {
repeat :打开新页面;
:执行页面处理;
backward: 处理下一个页面;
repeat while (还有待处理页面?)
}
stop
@enduml
</code></pre></div></div>
<p><img src="../../../../public/img/plantuml/e2.png" alt="流程图2" /></p>RK Fengplantuml使用总结Chrome85下有道云笔记插件异常2020-09-11T00:00:00+08:002020-09-11T00:00:00+08:00https://blog.arkfeng.xyz/2020/09/11/youdao_note_extension_chrome85<h1 id="chrome85下有道云笔记插件异常">Chrome85下有道云笔记插件异常</h1>
<p>Chrome 升级到 85.0.4183.102 版本后, 有道云笔记插件失效.</p>
<h2 id="有道云笔记插件无法打开chrome85">有道云笔记插件无法打开(chrome85)</h2>
<p>F12 调试, 点击有道云插件后, 报错</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Access to XMLHttpRequest at 'https://note.youdao.com/yws/mapi/user?method=get' from origin 'https://mp.weixin.qq.com' has been blocked by CORS policy:
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
</code></pre></div></div>
<p>怀疑是 chrome 85 新特性引起的, 搜索后找到官方文档: <a href="https://www.chromium.org/Home/chromium-security/extension-content-script-fetches">Changes to Cross-Origin Requests in Chrome Extension Content Scripts</a></p>
<p>经测试, 有效的设置为:</p>
<p>打开: <code class="language-plaintext highlighter-rouge">chrome://flags/#cors-for-content-scripts</code>, 设置为 <code class="language-plaintext highlighter-rouge">Disabled</code></p>
<h2 id="需要登录且收藏失败chrome85前">需要登录且收藏失败(chrome85前)</h2>
<p>该问题发生在 chrome85 版本前, 记录以前的解决方案.</p>
<p>参考: <a href="https://tieba.baidu.com/p/6549122973">Chrome80 网页剪报收藏笔记每次都需要登录且收藏失败报错</a></p>
<p>打开: <code class="language-plaintext highlighter-rouge">chrome://flags/#same-site-by-default-cookies</code>, 设置 为 <code class="language-plaintext highlighter-rouge">Disabled</code>.</p>
<h2 id="chrome85-下最终设置结果">chrome85 下最终设置结果</h2>
<p>效果图:</p>
<p><img src="../../../../public/img/chrome85_settings/chrome85_setting.png" alt="效果图" /></p>RK FengChrome85下有道云笔记插件异常python 动态加载所有子类2020-09-05T00:00:00+08:002020-09-05T00:00:00+08:00https://blog.arkfeng.xyz/2020/09/05/python_subclass<h1 id="python-动态加载所有子类">python 动态加载所有子类</h1>
<p>python 类提供 <code class="language-plaintext highlighter-rouge">__subclass__</code> 方法, 可用于获取所有子类列表. 同时, 提供 <code class="language-plaintext highlighter-rouge">issubclass</code>, 判断一个类是否是另一个类的子类.</p>
<h2 id="1-__subclass__-使用示例">1. <code class="language-plaintext highlighter-rouge">__subclass__</code> 使用示例</h2>
<p>使用方法见示例:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">unittest</span>
<span class="k">class</span> <span class="nc">TestSubClass</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">testSubClass</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">class</span> <span class="nc">A</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">B</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">C</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">for</span> <span class="n">sub_class</span> <span class="ow">in</span> <span class="n">A</span><span class="p">.</span><span class="n">__subclasses__</span><span class="p">():</span>
<span class="k">print</span><span class="p">(</span><span class="n">sub_class</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">issubclass</span><span class="p">(</span><span class="n">sub_class</span><span class="p">,</span> <span class="n">A</span><span class="p">))</span>
<span class="c1"># <class 'x.TestSubClass.testSubClass.<locals>.B'>
</span> <span class="c1"># <class 'x.TestSubClass.testSubClass.<locals>.C'>
</span>
<span class="k">def</span> <span class="nf">testSubClassV2</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">class</span> <span class="nc">A</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">B</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">C</span><span class="p">(</span><span class="n">B</span><span class="p">):</span>
<span class="k">pass</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">A</span><span class="p">.</span><span class="n">__subclasses__</span><span class="p">()),</span> <span class="mi">1</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">B</span><span class="p">.</span><span class="n">__subclasses__</span><span class="p">()),</span> <span class="mi">1</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">issubclass</span><span class="p">(</span><span class="n">B</span><span class="p">,</span> <span class="n">A</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">issubclass</span><span class="p">(</span><span class="n">C</span><span class="p">,</span> <span class="n">B</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">issubclass</span><span class="p">(</span><span class="n">C</span><span class="p">,</span> <span class="n">A</span><span class="p">))</span>
</code></pre></div></div>
<h2 id="2-使用-__subclass__-及-importlib-动态加载基类的所有实现类">2. 使用 <code class="language-plaintext highlighter-rouge">__subclass__</code> 及 <code class="language-plaintext highlighter-rouge">importlib</code> 动态加载基类的所有实现类</h2>
<p>示例:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">glob</span>
<span class="kn">import</span> <span class="nn">importlib.util</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">shutil</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="k">class</span> <span class="nc">TestDynamicImport</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">abspath</span><span class="p">(</span><span class="s">"./script"</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">prepare_scripts</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">prepare_scripts</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="k">def</span> <span class="nf">_create_script</span><span class="p">(</span><span class="n">_class_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">_script_file</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">content</span> <span class="o">=</span> <span class="s">"""
from collections import OrderedDict
class {class_name}(OrderedDict):
pass
"""</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">class_name</span><span class="o">=</span><span class="n">_class_name</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">_script_file</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">):</span>
<span class="n">shutil</span><span class="p">.</span><span class="n">rmtree</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">)</span>
<span class="n">os</span><span class="p">.</span><span class="n">makedirs</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="c1"># create SB
</span> <span class="n">_create_script</span><span class="p">(</span><span class="n">_class_name</span><span class="o">=</span><span class="s">"SB"</span><span class="p">,</span> <span class="n">_script_file</span><span class="o">=</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">,</span> <span class="s">"sb.py"</span><span class="p">))</span>
<span class="c1"># create SC
</span> <span class="n">_create_script</span><span class="p">(</span><span class="n">_class_name</span><span class="o">=</span><span class="s">"SC"</span><span class="p">,</span> <span class="n">_script_file</span><span class="o">=</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">,</span> <span class="s">"sc.py"</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">testDynamicImport</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" """</span>
<span class="c1"># 加载所有脚本
</span> <span class="n">py_file_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">py_file</span> <span class="k">for</span> <span class="n">py_file</span> <span class="ow">in</span> <span class="n">glob</span><span class="p">.</span><span class="n">glob</span><span class="p">(</span><span class="s">"{}/*.py"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">abspath</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">script_dir</span><span class="p">).</span><span class="n">rstrip</span><span class="p">(</span><span class="s">"/"</span><span class="p">)))]</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{} script found!"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">py_file_list</span><span class="p">)))</span>
<span class="k">for</span> <span class="n">py_file</span> <span class="ow">in</span> <span class="n">py_file_list</span><span class="p">:</span>
<span class="n">script_module_name</span> <span class="o">=</span> <span class="s">"scripts.{}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">py_file</span><span class="p">)[:</span><span class="o">-</span><span class="nb">len</span><span class="p">(</span><span class="s">".py"</span><span class="p">)])</span>
<span class="n">spec</span> <span class="o">=</span> <span class="n">importlib</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">spec_from_file_location</span><span class="p">(</span><span class="n">script_module_name</span><span class="p">,</span> <span class="n">py_file</span><span class="p">)</span>
<span class="n">script_module</span> <span class="o">=</span> <span class="n">importlib</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">module_from_spec</span><span class="p">(</span><span class="n">spec</span><span class="p">)</span>
<span class="n">sys</span><span class="p">.</span><span class="n">modules</span><span class="p">[</span><span class="n">script_module_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">script_module</span>
<span class="n">spec</span><span class="p">.</span><span class="n">loader</span><span class="p">.</span><span class="n">exec_module</span><span class="p">(</span><span class="n">script_module</span><span class="p">)</span>
<span class="c1"># 获取所有脚本类
</span> <span class="k">for</span> <span class="n">sub_class</span> <span class="ow">in</span> <span class="n">OrderedDict</span><span class="p">.</span><span class="n">__subclasses__</span><span class="p">():</span>
<span class="k">print</span><span class="p">(</span><span class="n">sub_class</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="nb">issubclass</span><span class="p">(</span><span class="n">sub_class</span><span class="p">,</span> <span class="n">OrderedDict</span><span class="p">))</span>
<span class="c1"># <class 'scripts.sb.SB'>
</span> <span class="c1"># <class 'scripts.sc.SC'>
</span></code></pre></div></div>RK Fengpython 动态加载所有子类python asyncio 使用总结(2)2020-09-04T00:00:00+08:002020-09-04T00:00:00+08:00https://blog.arkfeng.xyz/2020/09/04/python_asyncio_v2<h1 id="python-asyncio-使用总结2">python asyncio 使用总结(2)</h1>
<h2 id="1-使用线程池异步执行阻塞过程">1. 使用线程池异步执行阻塞过程</h2>
<p>由于所有协程过程是在同一个线程中执行的, 阻塞步骤会阻塞所有协程的执行.</p>
<p>官方文档<a href="https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools">Executing code in thread or process pools</a>, 推荐使用线程池执行阻塞步骤:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">concurrent.futures</span>
<span class="k">def</span> <span class="nf">blocking_io</span><span class="p">():</span>
<span class="c1"># File operations (such as logging) can block the
</span> <span class="c1"># event loop: run them in a thread pool.
</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'/dev/urandom'</span><span class="p">,</span> <span class="s">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">return</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">cpu_bound</span><span class="p">():</span>
<span class="c1"># CPU-bound operations will block the event loop:
</span> <span class="c1"># in general it is preferable to run them in a
</span> <span class="c1"># process pool.
</span> <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">i</span> <span class="o">*</span> <span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span> <span class="o">**</span> <span class="mi">7</span><span class="p">))</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">get_running_loop</span><span class="p">()</span>
<span class="c1">## Options:
</span>
<span class="c1"># 1. Run in the default loop's executor:
</span> <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">loop</span><span class="p">.</span><span class="n">run_in_executor</span><span class="p">(</span>
<span class="bp">None</span><span class="p">,</span> <span class="n">blocking_io</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'default thread pool'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="c1"># 2. Run in a custom thread pool:
</span> <span class="k">with</span> <span class="n">concurrent</span><span class="p">.</span><span class="n">futures</span><span class="p">.</span><span class="n">ThreadPoolExecutor</span><span class="p">()</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">loop</span><span class="p">.</span><span class="n">run_in_executor</span><span class="p">(</span>
<span class="n">pool</span><span class="p">,</span> <span class="n">blocking_io</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'custom thread pool'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="c1"># 3. Run in a custom process pool:
</span> <span class="k">with</span> <span class="n">concurrent</span><span class="p">.</span><span class="n">futures</span><span class="p">.</span><span class="n">ProcessPoolExecutor</span><span class="p">()</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">loop</span><span class="p">.</span><span class="n">run_in_executor</span><span class="p">(</span>
<span class="n">pool</span><span class="p">,</span> <span class="n">cpu_bound</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'custom process pool'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
</code></pre></div></div>RK Fengpython asyncio 使用总结(2)