• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            我要啦免费统计

            from http://docs.continuum.io/anaconda-cluster/examples/spark-caffe

            Deep Learning (Spark, Caffe, GPU)

            Description

            To demonstrate the capability of running a distributed job in PySpark using a GPU, this example uses a neural network library, Caffe. Below is a trivial example of using Caffe on a Spark cluster; although this is redundant, it demonstrates the capability of training neural networks with GPUs.

            For this example, we recommend the use of the AMI ami-2cbf3e44 and the instance type g2.2xlarge. An example profile (to be placed in ~/.acluster/profiles.d/gpu_profile.yaml) is shown below:

            name: gpu_profile
            node_id: ami-2cbf3e44 # Ubuntu 14.04 - IS HVM - Cuda 6.5
            user: ubuntu
            node_type: g2.2xlarge
            num_nodes: 3
            provider: aws
            plugins:
              - spark-yarn
              - notebook
            

            Download

            To execute this example, download the: spark-caffe.py example script or spark-caffe.ipynbexample notebook.

            Installation

            The Spark + YARN plugin can be installed on the cluster using the following command:

            $ acluster install spark-yarn
            

            Once the Spark + YARN plugin is installed, you can view the YARN UI in your browser using the following command:

            $ acluster open yarn
            

            Dependencies

            First, we need to bootstrap Caffe and its dependencies on all of the nodes. We provide a bash script that will install Caffe from source: bootstrap-caffe.sh. The following command can be used to upload the bootstrap-caffe.sh script to all of the nodes and execute it in parallel:

            $ acluster submit bootstrap-caffe.sh --all
            

            After a few minues, Caffe and its dependencies will be installed on the cluster nodes and the job can be started.

            Running the Job

            Here is the complete script to run the Spark + GPU with Caffe example in PySpark:

            # spark-caffe.py from pyspark import SparkConf from pyspark import SparkContext  conf = SparkConf() conf.setMaster('yarn-client') conf.setAppName('spark-caffe') sc = SparkContext(conf=conf)   def noop(x):     import socket     return socket.gethostname()  rdd = sc.parallelize(range(2), 2) hosts = rdd.map(noop).distinct().collect() print hosts   def caffe_process(x):     import os     os.environ['PATH'] = '/usr/local/cuda/bin' + ':' + os.environ['PATH']     os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda/lib64:/home/ubuntu/pombredanne-https-gitorious.org-mdb-mdb.git-9cc04f604f80/libraries/liblmdb'     import subprocess     proc = subprocess.Popen('cd /home/ubuntu/caffe && bash ./examples/mnist/train_lenet.sh', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)     out, err = proc.communicate()     return proc.returncode, out, err  rdd = sc.parallelize(range(2), 2) ret = rdd.map(caffe_process).distinct().collect() print ret 

            You can submit the script to the Spark cluster using the submit command.

            $ acluster submit spark-caffe.py 

            After the script completes, the trained Caffe model can be found at/home/ubuntu/caffe/examples/mnist/lenet_iter_10000.caffemodel on all of the compute nodes.

            posted on 2015-10-14 17:25 閱讀(3606) 評論(1)  編輯 收藏 引用 所屬分類: life關于人工智能的yy

            評論:
            # re: Deep Learning (Spark, Caffe, GPU) 2015-10-21 18:19 | 春秋十二月
            這是啥  回復  更多評論
              
            99久久99久久精品免费看蜜桃| 久久精品www| 久久精品国产亚洲AV忘忧草18| 欧美日韩久久中文字幕| 亚洲国产精品无码久久久久久曰 | 久久香蕉国产线看观看精品yw| 伊人久久大香线蕉av不变影院| 狠狠干狠狠久久| 狠狠色丁香久久婷婷综合图片| 日韩精品久久久久久免费| 热久久国产精品| 久久精品国产亚洲AV不卡| 久久综合久久综合久久| 香蕉久久久久久狠狠色| 色综合久久中文色婷婷| 亚洲国产成人精品久久久国产成人一区二区三区综 | 模特私拍国产精品久久| 久久精品九九亚洲精品天堂| 一本大道久久东京热无码AV | 99久久无码一区人妻| 成人久久免费网站| 久久精品人妻一区二区三区| 久久99国产综合精品免费| 久久久久99这里有精品10| 精品综合久久久久久88小说| 77777亚洲午夜久久多喷| 日韩人妻无码精品久久免费一| 日本亚洲色大成网站WWW久久| 亚洲国产天堂久久综合网站| 久久国产色AV免费看| 欧洲成人午夜精品无码区久久 | 色婷婷久久久SWAG精品| 国产精品伦理久久久久久| 亚洲国产精品人久久| 精品久久久无码中文字幕天天| 国产精品一久久香蕉国产线看观看 | 国产精品久久久久一区二区三区| av无码久久久久不卡免费网站| 婷婷综合久久中文字幕蜜桃三电影 | 少妇高潮惨叫久久久久久| 久久久久亚洲精品日久生情 |