Cloud Object Storage Service based on Openstack Prepared for 2011.05.03 KT 클라우드추진본부 /PEG 황진경
목차 I. OpenStack 클라우드서비스표준화 II. Object Storage 서비스구조및기술 III. 클라우드스토리지서비스제공사례 IV. 결언 hyomin@kt.com 2
I. OpenStack 클라우드서비스 표준화 jkhwang@kt.com 3
OpenStack (openstack.org) Public & Private 클라우드구축을위한오픈소스 software & 개발커뮤니티 Mission: 규모와상관없이 Public & Private 클라우드제공자들의요구사항들을만족시키고, implement 하기간단하며동시에 massively scalable 한유비쿼터스오픈소스클라우드플랫폼을제공한다. Why is OpenStack important? Open eliminates vendor lock-in Working together, we all go faster Freedom to federate, or move between clouds è 오픈소스기반클라우드표준지향 jkhwang@kt.com 4
OpenStack projects (code names) Compute (Nova): Provision and manage large networks of virtual machines Object Storage (Swift): Create petabytes of reliable storage using standard servers Image (Glance): Catalog and manage large libraries of server images (http + object storage) jkhwang@kt.com 5
OpenStack Community Snapshot jkhwang@kt.com 6 6
OpenStack Founding Principles Apache 2.0 license (OSI), NO 유료 엔터프라이즈 버전 Open design process, 수차례의공개 Design Summits 언제나접근가능한소스코드 repository 모든커뮤니티프로세스는문서화되고투명하게운영 Open Standard들을항상추구하고받아들임. 유연한 deployment를위한 Modular 디자인 (API) jkhwang@kt.com 7
OpenStack 구조의장점 Architect for inhouse Re-Architect for service provider Today s Reality Future with OpenStack Architect once Deploy anywhere jkhwang@kt.com 8
OpenStack 진행과정 2010 3 월 5 월 6 월 7 월 Rackspace 가자사의클라우드스토리지소프트웨어를오픈소스화하도록결정 NASA 가 Nebula 플랫폼을오픈소스화 Rackspace 와 NASA 가 OpenStack 을설립 Design Summit (Austin, TX) * Rackspace: IaaS 분야에서의 2 위사업자 * NASA Nebula: NASA 의서버가상화플랫폼 jkhwang@kt.com 9
OpenStack 진행과정 7 월 2011 10 월 11 월 2 월 25 개의파트너와함께 OpenStack 커뮤니티 Launch 첫번째공식버전릴리즈 (Austin Release) 35 파트너 첫번째 Public Design Summit (San Antonio, TX) 9 월 7 월 11 월 -12 월 2 월 두번째 Bexar 버전릴리즈 OpenStack 테스트베드구성 Follow-Up 시작 첫번째 Public Design Summit 참여 OpenStack 한국커뮤니티구축 OpenStack 공식파트너등록 jkhwang@kt.com 10
OpenStack 진행과정 April July 세번째 Cactus 코드릴리즈 Design Summit Santa Clara, CA 4 번째 Diablo 코드릴리즈계획 3 월 4 월말 5~6 월 OpenStack 기반클라우드스토리지서비스 CBT 두번째 Public Design Summit 참여 비즈니스세션발표 디자인서밋에코드제출 OpenStack 기반클라우드스토리지서비스 OBT~ 상용화 (ucloud SS) - OpenStack 저변확대를위한지속적인커뮤니티지원, 기술세미나, 컨퍼런스활동 - OpenStack 기반의서비스확장 jkhwang@kt.com 11
OpenStack Community 참여사 jkhwang@kt.com 12
OpenStack 을통한빌드업 Open, scalable 플랫폼으로시작 CLOUD OS OpenStack Compute OpenStack Object Storage OpenStack Image Service jkhwang@kt.com 13
OpenStack 을통한빌드업 User Ticketing Network Monitoring ECOSYSTEM Host Server Control Panel System Management Systems Management CLOUD OS OpenStack Compute OpenStack Object Storage OpenStack Image Service 필요한 3rd party 툴과솔루션들을에코시스템으로부터수용 jkhwang@kt.com 14
OpenStack 을통한빌드업 User Ticketing Network Monitoring ECOSYSTEM Host Server Control Panel System Management Systems Management CLOUD OS OpenStack Compute OpenStack Object Storage OpenStack Image Service Account Billing Admin CLI Tools Live Chat Support Account Management PUBLIC CLOUD Customer facing & admin services for public cloud jkhwang@kt.com 15
OpenStack 을통한빌드업 User Ticketing Network Monitoring ECOSYSTEM Host Server Control Panel System Management Systems Management CLOUD OS OpenStack Compute OpenStack Object Storage OpenStack Image Service Admin Control Panel Dept. Accounting Chargeback User Management Enterprise Software Integration Systems PRIVATE CLOUD Integrate with existing enterprise systems forprivate jkhwang@kt.com cloud 16
II. Object Storage 서비스구조 및기술 jkhwang@kt.com 17
Simple Storage Service 개념 온라인웹서비스로제공되는스토리지서비스 크기제한없이사용가능하고, 쓴만큼만과금 스토리지가상화기술 : 물리저장시스템의위치나크기에관계없이, 이를논리적인 ( 하나의 ) 스토리지로매핑 ( 추상화 ) 하여할당, 사용, 관리할수있도록하는기술 간단한 RESTful API 제공 GET, DELETE, PUT, COPY, HEAD, POST 웹호스팅, 이미지호스팅, 대용량데이터의백업용도 Amazon S3 (Simple Storage Service) 와유사 KT UCLOUD SS hyomin@kt.com 18
OpenStack Object Storage OpenStack Object Storage (code name: Swift) Highly available, distributed, eventually consistent 1 object/blob 2 store Petabyte 급 storage 의 clustering and management software architecture commodity server 기반저렴하고안정적인대규모스토리지서비스 객체들이여러클러스터내하드웨어장치에분산중복저장되고이들간에무결성을제공. 새로운노드가추가되면자동적으로구성되고, 오류발생에대비하여타액티브노드에중복 Eventually consistent: 분산시스템에서, 중복데이터중하나를수정시, 즉시반영하지않고차후에처리하여, 결국은일관성을유지하도록하는효율화개념 Blob (binary large object): 이미지와같은디지털데이터로서비정형적, 큰크기의객체 19 hyomin@kt.com 19
OpenStack Object Storage Swift 특징 hyomin@kt.com 20
OpenStack Object Storage 구조및기능 Proxy Server: 사용자의요청에따라알맞은서버에연결하여서비스를제공 Account Server: 사용자계정관리및계정별컨테이너조회 Container Server: 사용자계정의컨테이너를관리하는서버. 계정하의컨테이너들을관리하며컨테이너가가지고있는오브젝트들을조회 Object Server: 컨테이너내의오브젝트들을관리하는서버. 각각의오브젝트를실질적으로저장하며조회 hyomin@kt.com 21
OpenStack Object Storage Account, Container, Object 개념 Account 사용자를대표하는 URL 을할당받는 user 그룹 Container 들을관리하는객체 Account 안에는여러 user 가존재가능하며 사용자는 Account:user 를아이디로서암호를갖고 token 으로서구분가능 http://swift.kt.com/v1/auth_57069e6e-2b54-4eeb-95a9-1eff53637978 Container Container 폴더와같이 Object 를관리하는객체 (ex. Amazon S3 의 Bucket) user 들은 Access Control List(ACL) 를통해 Container 의읽기쓰기권한을관리 http://swift.kt.com/v1/auth_57069e6e-2b54-4eeb-95a9-1eff53637978/testcontainer 실질적으로저장되는파일오브젝트 Object Object Container 의 ACL 에적용되어관리 Object 는파일뿐만이아닌폴더형태로도저장가능 ( 즉, 폴더구조가능 ) http://swift.kt.com/v1/auth_57069e6e-2b54-4eeb-95a9-1eff53637978/testcontainer/testfile.txt - 22 - hyomin@kt.com 22
OpenStack Object Storage 데이터접근방법 Proxy Server Account Ring Object Ring Container Ring http://swift.kt.com/v1/account http://swift.kt.com/v1/account/container/object http://swift.kt.com/v1/account/container Account Server Container Serverr Object Server Account DB Container DB Object File - 23 - hyomin@kt.com 23
OpenStack Object Storage 사이트 : http://swift.openstack.org swift 설치및 administrator, developers guide 문서 소스코드 : http://launchpad.net/swift 버전업에따라업데이트 현재 : Swift-1.3.0 (cactus) Python 2.6 Ubuntu 10.04 xfs 파일시스템 (xattr 파일포맷 ) Sqlite3 DB hyomin@kt.com 24
자동디플로이먼트 대규모서버설정및설치를자동화 Hardware Setting OS Setting Software Setting clean hardware Swift ready HW 물리적설정 MAC MAC or IP IP IP OS Image Role Kickstart clean hardware`s o MAC o IP o OS o chef roles o (host name, domain name) hyomin@kt.com 25
III. 클라우드스토리지서비스 제공사례 jkhwang@kt.com 26
KT ucloud SS (storage service) 클라우드스토리지서비스오픈 Closed Beta Service : 3.28 ~ 4.22 Open Beta: soon to be open cs.ucloud.com/ss 5월중오픈베타에서만나보세요 ^^ * cs: compute service * ss: storage service hyomin@kt.com 27
OpenStack Swift 기반응용들 (blog) Here are some good use cases for OpenStack Object Storage: Storing media libraries (photos, music, videos, etc.)- 다운로드사이트 Archiving video surveillance files Archiving phone call audio recordings Archiving compressed log files Archiving backups (<5GB each object) Storing and loading of OS Images, etc. Storing file populations that grow continuously on a practically infinite basis. Storing small files (<50 KB). OpenStack Object Storage is great at this. Storing billions of files. Storing Petabytes (millions of Gigabytes) of data. jkhwang@kt.com 28 http://adrianotto.com/2010/09/openstack-os-is-great-for/
Amazon / Rackspace 서비스예시 Amazon S3/EC2 Application Hosting Backup and Storage Content Delivery E-Commerce High Performance Computing Media Hosting On-Demand Workforce Search Engines Web Hosting Rackspace Cloudfiles Email Hosting Email Archiving Email marketing File Sharing Backup & Collaboration Admin Tools control panel, migration app Mobile applications http://aws.amazon.com/solutions/case-studies/ http://69.20.89.3/apps/backup_and_collabor ation/online_file_storage/# jkhwang@kt.com 29
Rackspace API 서비스예시 Rackspace CloudFiles API 활용을통한응용예 각응용개발자들이, API 활용을통한응용제공의예 CyberDuck : 파일전송프로그램 Mobile App: 모바일단말에서파일접근 (CDN) Plixi(Tweet Photo): 트위터업로드사진저장 http://www.rackspace.com/cloud/cloud_hosting_products/files/api/ jkhwang@kt.com 30
Amazon S3 mashup API 서비스예시 jkhwang@kt.com 31
OpenStack Swift API 기반클라이언트툴 KT ucloud SS (Swift) 와연동가능한 tools 도구이름 설명 URL 연동 Cloud storage Cyberduck Open source FTP, SFTP, WebDAV, Cloud Files, Google Docs & Amazon S3 Browser for Mac & Windows. Cyberduck.ch Amazon S3 FTP, WebDav Google Docs Cloudfiles Window Azure Openstack Etc Gladinet Seamless access, aggregate, and backup to cloud storage Gladinet.com Amazon S3 FTP, WebDav Google Docs Cloudfiles Internap Openstack Etc Cloudfuse Cloudfuse is a FUSE application which provides access to Rackspace's Cloud Files (or any installation of Swift). ohloh.net/p/cloud fuse Cloudfiles Openstack(Sw ift) jkhwang@kt.com 32
Swift API 종류 API 설명 Account and Authentication Storage account service Storage container service Storage object service GET account GET accounts list PUT account DELETE account GET user GET users list PUT user DELETE user GET Storage account HEAD Storage account POST Storage account HEAD Storage container GET Storage container PUT Storage container DELETE Storage container POST Storage container : user metadata POST Storage container : ACL HEAD Storage object GET Storage object PUT Storage object Chunked Transfer Encoding Copy Object POST Storage object DELETE Storage object Account 생성, 조회, 삭제 User 생성, 조회, 삭제사용자인증처리 (token, storage url) Container 리스트, 수량조회 Account 전체사용량조회 Account metadata 추가 Object 리스트, 수량조회 Container 전체사용량조회 Container 생성, 삭제 Container metadata 추가 Container ACL 설정 Object 정보조회 Object 업로드및다운로드 Object copy Object 삭제 Object metadata 추가 jkhwang@kt.com 33
Swift 사용절차 1. 인증절차 : ( ID,PW ) 를통해토큰획득 2. 요청 (GET,PUT,..) with 토큰값 ID: test:tester PW: testing 인증 Client ID, Pass 전송 > GET /auth/v1.0 HTTP/1.1 > X-Storage-User: test:tester > X-Storage-Pass: testing URL, Token 전송 < HTTP/1.1 200 OK < X-Storage-Url: http://127.0.0.1:8080/v1/auth_b54c7309-4fe2-496d-aeb4-496a564017a8 < X-Storage-Token: AUTH_tk938f7f8746974c91b2992faa95ddca73 Swauth middleware Proxy server Request 전송 > GET /v1/auth_b54c7309-4fe2-496d-aeb4-496a564017a8 HTTP/1.1 > X-Storage-Token: AUTH_tk938f7f8746974c91b2992faa95ddca73 일반적인 request ACL 확인 A,C,O 서버처리 Response 전송 < HTTP/1.1 204 No Content jkhwang@kt.com 34
Swift 일정 Swift 1.0.2 Default Swift API (GET, PUT, DELETE, COPY) Upload Files, Download Files, Delete Files Create Containers, Delete Containers etc. Swift 1.1 (Austin) Access Control List(ACL) & Public Container User-definable Metadata for Accounts and Containers Statistics for Object Storage Swift 1.2 (Bexar) Multi region replication Incorporate SW-auth into Swift S3 API Middleware Swift 1.3 (Cactus) Multi region asynchronous-proxy checksum-get SQLite write ahead logging Swift 1.4 (Diablo) 컨테이너에서 index.html 직접접근표시 자주접근하는파일에대해빠른에러검출 빈번한요청경우성능개선 (proxy refactoring) SQLite 인덱싱성능개선for jkhwang@kt.com 컨테이너, 35 어카운트, 인증개선, 내부과금과통합
IV. Summary 오픈소스프로젝트를통한실질적인클라우드표준화접근 멤버수가빠르게증가. 주요벤더, 사업자참여 완성도가높은 object storage (swift) 기반서비스빌드사례소개및기술소개 à KT ucloud SS 서비스 앞으로발전추세이며국내멤버의참여기대 compute (NOVA) 도곧상용수준완성도기대. image (Glance) 서비스프로젝트외다음프로젝트시작예정 burrow : simple queuing service loadbalancing service database service 한국오픈스택커뮤니티 : openstack.or.kr 많은참여부탁드립니다! jkhwang@kt.com 36
cs.ucloud.com 감사합니다 jkhwang@kt.com 37